Image processing device, image processing method, image processing program, and image display device

ABSTRACT

Provided is an image processing device capable of performing appropriate noise removal or reduction on input images of various resolution. The image processing device includes a contour direction estimating unit ( 21 ) that estimates a contour direction in which a signal value is a constant value for each pixel, a low pass filter unit ( 20   a ) that smoothes the signal value of the pixel based on the signal value of each reference pixel serving as a pixel of a reference region according to the pixel and arranged in the contour direction of the pixel estimated by the contour direction estimating unit for each pixel, and a parameter deciding unit ( 20   b ) that decides intensity of smoothing by the low pass filter unit according to an enlargement factor obtained based on a ratio of resolution of an input image signal and resolution of an output image signal.

TECHNICAL FIELD

The present invention relates to an image processing device thatperforms image conversion according to resolution.

BACKGROUND ART

There is a demand for effectively utilizing image content by enablingimage content with the spread of an information communication technologyto be viewed under a condition different from a condition when created.For example, in an Internet protocol television (IPTV) or the like,there are cases in which image content (a so-called network movingimage) of relatively low resolution that is originally created to beviewed through a mobile terminal device or the like is displayed on animage display device of high resolution. Here, image content havingresolution of 640 pixels (a horizontal direction)×360 pixels (a verticaldirection) is displayed on a display that supports a full highdefinition (HD) standard of 1920 pixels (the horizontal direction)×1080pixels (the vertical direction). In this case, resolution may beconverted into high resolution by interpolating a signal value of eachpixel included in image content between pixels (this is also referred toas up-scaling or up-convert).

There are cases in which the following noises are remarkably shown in animage whose resolution is increased:

(1) jaggy: a step-like contour shown in a slant line or a curved line;

(2) mosquito noise: wave-like noise shown in a portion in which acontrasting density or color signal value abruptly changes or itsvicinity when a compression-coded code is decoded; and

(3) dot interference: noise shown in a granular form in a boundary inwhich a color changes when separation (YC separation) of a brightnesssignal and a color-difference signal from a composite signal isinappropriate.

In this regard, in the past, an attempt to cause a signal valueinterpolated into an image having an increased resolution to be subjectto a low pass filter and reduce noise has been made. For example, in aprocessing device disclosed in JP 2005-353068 A, a window of a certainsize is set on an input current frame/field based on a current pixel, aneigenvalue and an eigenvector used to determine characteristics of thewindow are calculated, the characteristics of the window are determinedbased on the calculated eigenvalue, a filtering weighted value to beapplied is decided based on the determination result, and filtering isperformed on the window based on the calculated eigenvector and thedecided filtering weighted value.

In addition, an attempt to perform image sharpening (which is alsoreferred to as “emphasis/enhancement”) by emphasizing information of ahigh frequency component of an interpolated signal value has been made.

Further, an image processing device that performs image conversionaccording to a type of a broadcast wave of video has been proposed. Forexample, an image processing device disclosed in JP 2010-93856 Aincludes an image converting unit that converts a first video signalhaving first resolution into a second video signal having secondresolution higher than the first resolution according to a parameterindicating a ratio of pixels of a high frequency component to beinserted into pixels of the first video signal and a control unit thatperforms control such that the parameter is changed according to a typeof a broadcast wave of the first video signal.

SUMMARY OF INVENTION Technical Problem

As described above, it is effective to perform a noise reduction processand an image sharpening process on an image having resolution. However,when the noise reduction process and the image sharpening process areperformed on an image that is originally high in resolution and low inan interpolation process level or an image that has not undergone aninterpolation process, an effect is reduced, and an artifact occursinstead.

Solution to Problem

The present invention was made in light of the foregoing, and it isdesirable to provide an image processing device capable of performingappropriate noise removal or reduction on input images of variousresolution.

An image processing device disclosed herein includes: a contourdirection estimating unit that estimates a contour direction in which asignal value is a constant value for each pixel; a low pass filter unitthat smoothes the signal value of the pixel based on the signal value ofeach reference pixel serving as a pixel of a reference region accordingto the pixel and arranged in the contour direction of the pixelestimated by the contour direction estimating unit for each pixel; and aparameter deciding unit that decides intensity of smoothing by the lowpass filter unit according to an enlargement factor obtained based on aratio of resolution of an input image signal and resolution of an outputimage signal.

Advantageous Effects of Invention

According to the present invention, it is possible to performappropriate noise removal or reduction on input images of variousresolution.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram illustrating a configuration of a displaydevice according to a first embodiment of the invention.

FIG. 2 is a schematic diagram illustrating a configuration of an imageprocessing unit according to the present embodiment.

FIG. 3 is a conceptual diagram illustrating a configuration of a highfrequency expanding unit according to the present embodiment.

FIG. 4 is a schematic diagram illustrating an exemplary configuration ofa high frequency expanding unit according to the present embodiment.

FIG. 5 is a schematic diagram illustrating a configuration of a verticalhigh pass filter unit according to the present embodiment.

FIG. 6 is a schematic diagram illustrating a configuration of ahorizontal high pass filter unit according to the present embodiment.

FIG. 7 is a schematic diagram illustrating an exemplary configuration ofa non-linear operation unit according to the present embodiment.

FIG. 8 is a schematic diagram illustrating another exemplaryconfiguration of a non-linear operation unit according to the presentembodiment.

FIG. 9 is a conceptual diagram illustrating an exemplary contour.

FIG. 10 is a conceptual diagram illustrating an exemplary referenceregion.

FIG. 11 is a conceptual diagram illustrating an exemplary x directiondifferential filter.

FIG. 12 is a conceptual diagram illustrating an exemplary y directiondifferential filter.

FIG. 13 is a conceptual diagram illustrating an exemplary x directionpartial differential.

FIG. 14 is a conceptual diagram illustrating an exemplary y directionpartial differential.

FIG. 15 illustrates an exemplary quantization contour directioncandidate.

FIG. 16 is a conceptual diagram illustrating an exemplary quantizationcontour direction calculation.

FIG. 17 is a conceptual diagram illustrating an exemplary referenceregion weighting.

FIG. 18 is a conceptual diagram illustrating another exemplary referenceregion weighting.

FIG. 19 is a conceptual diagram illustrating an exemplary directionevaluation value.

FIG. 20 is a conceptual diagram illustrating another exemplary directionevaluation value.

FIG. 21 is a conceptual diagram illustrating an exemplary directionevaluation region weighting.

FIG. 22 is a conceptual diagram illustrating another exemplary directionevaluation region weighting.

FIG. 23 is a conceptual diagram illustrating an exemplary smoothingtarget pixel.

FIG. 24 is a flowchart illustrating image processing according to thepresent embodiment.

FIG. 25 is a schematic diagram illustrating a configuration of a displaydevice according to a second embodiment of the present invention.

FIG. 26 is a diagram schematically illustrating a relation between anenlargement factor and intensity of processing by a low pass filter unitand a high frequency expanding unit.

FIG. 27 is a schematic diagram illustrating a configuration of a displaydevice according to a third embodiment of the present invention.

FIG. 28 is a schematic diagram illustrating a configuration of an imageprocessing unit according to a first modified example of the presentembodiment.

FIG. 29 is a flowchart illustrating image processing according to thepresent modified example.

FIG. 30 is a schematic diagram illustrating a configuration of an imageprocessing unit according to a second modified example of the presentembodiment.

FIG. 31 is a flowchart illustrating image processing according to thepresent modified example.

FIG. 32 is a schematic diagram illustrating a configuration of a highfrequency expanding unit according to a third modified example of thepresent embodiment.

FIG. 33 is a schematic diagram illustrating a configuration of a 2D highpass filter unit according to the present modified example.

FIG. 34 is a conceptual diagram illustrating an exemplary selectionreference pixel.

FIG. 35 is a flowchart illustrating image processing according to thepresent modified example.

FIG. 36 illustrates an example of an image related to a brightnesssignal before and after processing is performed.

FIG. 37 illustrates an example of a spatial frequency characteristic ofan image before and after processing is performed.

FIG. 38 illustrates a spatial frequency characteristic of data used orgenerated in the present embodiment.

FIG. 39 is a diagram illustrating an exemplary output frequency by anon-linear operation.

FIG. 40 is a diagram illustrating another exemplary output frequency bya non-linear operation.

MODE FOR CARRYING OUT THE INVENTION

An image processing device according to an embodiment of the presentinvention includes a contour direction estimating unit that estimates acontour direction in which a signal value is a constant value for eachpixel, a low pass filter unit that smoothes the signal value of thepixel based on the signal value of each reference pixel serving as apixel of a reference region according to the pixel and arranged in thecontour direction of the pixel estimated by the contour directionestimating unit for each pixel, and a parameter deciding unit thatdecides intensity of smoothing by the low pass filter unit according toan enlargement factor obtained based on a ratio of resolution of aninput image signal and resolution of an output image signal (a firstconfiguration).

According to the above configuration, the signal value of each pixel issmoothed in the contour direction, and a noise in the contour directionwhich is visually sensitive is removed or reduced. Further, according tothe above configuration, the image processing device can changeintensity of smoothing according to the enlargement factor.

In the first configuration, a plurality of input terminals to which theinput image signal is input may be further provided, and the parameterdeciding unit may determine the resolution of the input image signalbased on a type of the input terminal to which the input image signal isinput among the plurality of input terminals (a second configuration).

In the first configuration, a scaling unit that obtains a resolutionratio serving as the ratio of the resolution of the input image signaland the resolution of the output image signal, performs interpolation ofthe input image based on the resolution ratio, and outputs theresolution ratio to the parameter deciding unit may be further provided,and the parameter deciding unit may use the resolution ratio as theenlargement factor (a third configuration).

In the first configuration, a plurality of input terminals to which theinput image signal is input and a scaling unit that obtains a resolutionratio serving as the ratio of the resolution of the input image signaland the resolution of the output image signal, performs interpolation ofthe input image based on the resolution ratio, and outputs theresolution ratio to the parameter deciding unit may be further provided,and the parameter deciding unit may determine the resolution of theinput image signal based on a type of the input terminal when theresolution of the input image signal is determined to be unique based onthe type of the input terminal to which the input image signal is inputamong the plurality of input terminals and use the resolution ratio asthe enlargement factor when the resolution of the input image signal isdetermined to be unique based on the type of the input terminal to whichthe input image signal is input among the plurality of input terminals(a fourth configuration).

In any one of the first to fourth configurations, preferably, a highfrequency expanding unit that generates a high frequency component ofthe signal value of the pixel for each pixel, and expands a frequencyband for the signal value of the pixel is further provided, and theparameter deciding unit decides intensity of processing by the highfrequency expanding unit according to the enlargement factor (a fifthconfiguration).

According to the above configuration, the high frequency component ofthe smoothed signal value is compensated by the high frequency expandingunit. Thus, it is possible to sharpen an image while avoiding orsuppressing a noise caused by aliasing. Further, according to the aboveconfiguration, the image processing device can change intensity ofsharpness according to the enlargement factor.

EMBODIMENTS

Hereinafter, embodiments will be described with reference to theappended drawings.

First Embodiment

FIG. 1 is a schematic diagram illustrating a configuration of a displaydevice 1 according to a first embodiment of the invention.

The display device 1 includes input terminals 10 a to 10 d, an inputunit 11, a scaling unit 13, an image processing unit 20, an image formatconverting unit 14, and a display unit 15.

Signals are input from the outside to the input terminals 10 a to 10 d.Examples of the signals input to the input terminals 10 a to 10 dinclude a composite video signal, a high frequency signal related toanalog television broadcasting, a high frequency signal related todigital television broadcasting, and a signal related to high-definitionmultimedia interface (HDMI).

The input unit 11 includes signal processing units 11A to 11D accordingto types of signals input to the input terminals 10 a to 10 d. Each ofthe signal processing units 11A to 11D converts the input signal into acertain image signal including a digital brightness signal Y, a digitalcolor-difference signal Cb, and a digital color-difference signal Cr.

For example, the signal processing unit 11B extracts a modulation signalrelated to a channel designated from the high frequency signal inputfrom the antenna (analog), and converts the extracted modulation signalinto a modulation signal of a base frequency band. The signal processingunit 11B demodulates the converted modulation signal to generate animage signal, and separates the analog brightness signal Y, the analogcolor-difference signal Cb, and the analog color-difference signal Crfrom the generated image signal. The signal processing unit 11B convertsthe separated signals from the analog signals to digital signals at apredetermined sampling frequency.

The input unit 11 selects one of a plurality of signals input to theinput unit 11, converts the input signal through one of the signalprocessing units 11A to 11D, and outputs the converted signal to thescaling unit 13. At this time, the input unit 11 outputs a signal Ss ofidentifying the selected input signal to the image processing unit 20.

When the resolution (the number of pixels) of the image signal inputfrom the input unit 11 is different from the resolution of the displayunit 15, the scaling unit 13 adjusts (scales) the resolution of theinput image signal so that the resolution of the image signal is equalto the resolution of the display unit 15. When the resolution of thedisplay unit 15 is higher than the resolution of the input image, thescaling unit 13 performs interpolation on the input image signal. Whenthe resolution of the display unit 15 is lower than the resolution ofthe input image, the scaling unit 13 performs down sampling on the inputimage signal. For example, the scaling unit 13 uses a scheme such as abicubic technique or a bilinear technique as a scheme for interpolationor down sampling. The scaling unit 13 outputs the image signal havingthe adjusted resolution to the image processing unit 20. When theresolution of the input image signal is equal to the resolution of thedisplay unit 15, the input image signal is output to the imageprocessing unit 20.

In the following description, a ratio of the number of pixels of thedisplay unit 15 in the horizontal direction (or the vertical direction)to the number of pixels of the input image signal in the horizontaldirection (or the vertical direction) is referred to as an enlargementfactor. For example, when the resolution of the input image signal is640 pixels (the horizontal direction)×360 pixels (the verticaldirection), and the resolution of the display unit 15 is 1920 pixels(the horizontal direction)×1080 pixels (the vertical direction), theenlargement factor is 3.

The image processing unit 20 performs processing related to noisereduction and image sharpening on the brightness signal Y among theimage signals input from the scaling unit 13, and generates a brightnesssignal Z indicating an image obtained by the noise reduction and thesharpening. The image processing unit 20 updates the brightness signal Yinput from the scaling unit 13 to the generated brightness signal Z, andsynchronizes the brightness signal Z with the color-difference signalsCb and Cr. The image processing unit 20 outputs an image signalincluding the brightness signal Y″ and the color-difference signals Cband Cr to the image format converting unit 14. A configuration of theimage processing unit 20 will be described later.

The image format converting unit 14 converts a format of the imagesignal input from the image processing unit 20. When the input imagesignal is an interlace signal, the image format converting unit 14converts the format of the image signal into a progressive signal. Theinterlace signal is a signal obtained by scanning pixels in every othercolumns in the horizontal direction of the pixel, and a signal in whicha scanning target column differs according to a frame. The progressivesignal is a signal obtained by scanning pixels in every column in thehorizontal direction of the pixel. The image format converting unit 14converts the input image signal or the image signal having the convertedformat into an image signal (for example, an RGB signal: an image signalincluding signal values of red (R), green (G), and blue (B) colors)represented by a color system supported by the display unit 15, andoutputs the converted image signal to the display unit 15.

The display unit 15 displays an image indicated by the image signalinput from the image format converting unit 14. For example, the displayunit 15 is a liquid crystal display (LCD) of a full high definition(which is also referred to as full HD) scheme, that is, an LCD havingresolution of 1920 pixels (the horizontal direction)×1080 pixels (thevertical direction). The display unit 15 includes pixel elements of red,green, and blue colors that are arranged two-dimensionally. As a result,the display unit 15 emits, for example, backlight light through thepixel elements at a brightness according to signal values of the pixelsindicated by the input image signal, and displays a color image.

(Configuration of Image Processing Unit 20)

Next, a configuration of the image processing unit 20 will be described.

FIG. 2 is a schematic diagram illustrating a configuration of the imageprocessing unit 20 according to the present embodiment.

The image processing unit 20 is configured to include a contourdirection estimating unit 21, a low pass filter unit 20 a, a parameterdeciding unit 20 b, and a high frequency expanding unit 27.

The contour direction estimating unit 21 estimates the contour directionfor each pixel based on the signal value (the brightness value) of eachpixel. The low pass filter unit 20 a filters the signal value of eachpixel using the signal value of each reference pixel that is arranged inthe contour direction of each pixel estimated by the contour directionestimating unit 21 and in the predetermined reference region from eachpixel.

The high frequency expanding unit 27 generates a high frequencycomponent of the signal value of each pixel filtered by the low passfilter unit 20 a, and expands the frequency band of the signal value ofeach pixel.

The parameter deciding unit 20 b decides a parameter Pm and a parameterPn based on the signal Ss. The parameter deciding unit 20 b outputs theparameter Pm to the low pass filter unit 20 a, and outputs the parameterPn to the high frequency expanding unit 27.

The parameter Pm is a parameter of deciding a degree in which an effectof processing by the low pass filter unit 20 a is reflected. Forexample, the parameter Pm has a value of 0≦Pm≦1. As the value of Pm getscloser to 1, the effect of processing by the low pass filter unit 20 ais increased, and when Pm=0, the effect of processing by the low passfilter unit 20 a is not reflected.

The parameter Pn is a parameter of deciding a degree in which an effectof processing by the high frequency expanding unit 27 is reflected. Forexample, the parameter Pn has a value of 0≦Pn≦1. As the value of Pn getscloser to 1, the effect of processing by the high frequency expandingunit 27 is increased, and when Pn=0, the effect of processing by thehigh frequency expanding unit 27 is not reflected.

The parameter deciding unit 20 b decides a value that gets closer to 1as the parameter Pm and the parameter Pn as the enlargement factor inthe scaling is increased. In other words, the parameter Pm and theparameter Pn are decided such that when the enlargement factor is large,the effects of processing by the low pass filter unit 20 a and the highfrequency expanding unit 27 are strongly reflected, whereas when theenlargement factor is small, the effects of processing by the low passfilter unit 20 a and the high frequency expanding unit 27 are lessreflected. Here, even when the enlargement factor is 1 (when theinterpolation and the like are not performed), it is desirable to decidea larger value than 0 as the parameter Pm and the parameter Pn in orderto enhance by removing a fine noise.

In the present embodiment, the parameter deciding unit 20 b decides theparameters Pm and Pn based on the signal Ss input from the input unit 11to the parameter deciding unit 20 b. More specifically, the parameterdeciding unit 20 b includes a memory (not illustrated). A table of thevalue of the parameter Pm according to the signal Ss and the value ofthe parameter Pn according to the signal Ss which are set in advance isstored in the memory. The parameter deciding unit 20 b decides theparameter Pm and the parameter Pn with reference to the table.

An example of a relation among an input terminal (terminal), inputresolution, and intensity of processing (processing intensity) by thelow pass filter unit 20 a and the high frequency expanding unit 27 isshown in Table 1.

TABLE 1 Processing Terminal Input resolution intensity Composite SDStrong Antenna (analog) SD Strong Antenna (digital) SD weak HDMI SD toHD weak

The resolution of input signals from the composite terminal and theantenna (analog) terminal is a standard definition (SD) of 720×480pixels. Further, it is possible to determine whether an input signalfrom the antenna terminal is an analog signal or a digital signal in atuner. When the resolution of the display unit 15 is HD, the enlargementfactor is 2.67 or 2.25. The parameter deciding unit 20 b decides theparameters so that processing by the low pass filter unit 20 a and thehigh frequency expanding unit 27 is enhanced in order to reduce a noisesuch as a jaggy associated with scaling (up-scaling) from SD to HD.Specifically, a certain value (for example, 0.9) close to 1 is set tothe table stored in the memory of the parameter deciding unit 20 b asthe values of the parameter Pm and Pn for the composite terminal and theantenna (analog) terminal.

The resolution of an input signal from the antenna (digital) terminal isHD. When the resolution of the display unit 15 is HD, the enlargementfactor is 1. When the enlargement factor is 1 or when the enlargementfactor is close to 1, a noise generated by up-scaling is small. When theeffects of processing by the low pass filter unit 20 a and the highfrequency expanding unit 27 are strongly reflected on the image, it maycause an artifact contrarily. For this reason, the parameter decidingunit 20 b decides the parameters so that intensity of processing by thelow pass filter unit 20 a and the high frequency expanding unit 27 isdecreased. Specifically, for example, a certain value (for example, 0.9)close to 1 is set to the table stored in the memory of the parameterdeciding unit 20 b as the values of the parameter Pm and Pn for thecomposite terminal and the antenna (analog) terminal.

An input signal from the HDMI terminal may have a wide range ofresolution of SD to HD. As described above, processing by the low passfilter unit 20 a and the high frequency expanding unit 27 is effectivein removal of a noise associated with up-scaling but may cause anartifact when the enlargement factor is 1 or close to 1. Since theoccurrence of an artifact is more undesirable than when a noise remains,in the example of Table 1, the parameter deciding unit 20 b decides theparameters so that intensity of processing by the low pass filter unit20 a and the high frequency expanding unit 27 is decreased.

For example, the parameter deciding unit 20 b may be configured todecide different values as the parameter Pm and the parameter Pn inorder to obtain a desired image processing effect. For example, in orderto generate an image in which a sense of fineness is strong felt, theparameter Pm may be set to a value smaller than the parameter Pn in thetable stored in the memory of the parameter deciding unit 20 b.

The low pass filter unit 20 a is configured to include a directionevaluating unit 22, a reference region weighting processing unit 23, apre-processing unit 24, a product-sum operation unit 25, and acomposition operation unit 26.

The contour direction estimating unit 21 estimates a contour directionof each pixel based on a signal value of each pixel indicated by thebrightness signal Y input from the scaling unit 13. The contourdirection refers to a direction perpendicular to a normal line of a lineserving as a contour, that is, a tangential direction of a line servingas a contour. A line serving as a contour represents a light indicatinga space in which a signal value is substantially constant, and may be acurved line or a straight line. Thus, a contour is not limited to aregion in which a signal value changes abruptly according to a change ina position. A relation between a line serving as a contour and a signalvalue corresponds to a relation between a contour line and an altitude.Since a position of each pixel is given discretely or influenced bynoise around a contour serving as an improvement target in the inventionsuch as jaggy, dot interference, and mosquito noise, there are cases inwhich it is difficult to decide a contour direction using a line passingbetween pixels having constant signal values as a line serving as acontour. Here, a signal value is assumed to be differentiable (that is,continuous) in a space representing coordinates of each pixel. Thecontour direction estimating unit 21 calculates a contour direction θbased on a differential value of a signal value in the horizontaldirection or the vertical direction, for example, based on Formula (1)for each pixel.[Mathematical Formula 1]θ=tan⁻¹(−[∂Y(x,y)/∂x]/[∂Y(x,y)/∂y]  (1)

In Formula (1), the contour direction θ is a counterclockwise anglebased on the horizontal direction (the x direction). x and y arecoordinates in the horizontal direction and the vertical direction,respectively. Y(x,y) is a signal value at the coordinates (x,y). Inother words, the contour direction θ is calculated as an angle providinga tangent value obtained by dividing a partial differential of thesignal value Y(x,y) in the x direction by a partial differential of thesignal value Y(x,y) in the y direction. Formula (1) can be derived froma relation in which the signal value Y(x,y) is constant although thecoordinates (x,y) are different. Here, G_(x)(x,y) and G_(y)(x,y)indicate the partial differential of the signal value Y(x,y) in the xdirection and the partial differential of the signal value Y(x,y) in they direction, respectively. In the following description, G_(x)(x,y) andG_(y) (x,y) are also referred to as an x direction partial differentialand a y direction partial differential, respectively.

Unless otherwise set forth in the following description, a position(coordinates) of a pixel (i,j) indicates a center of gravity point ofthe pixel. A variable a in the position of the pixel is indicated bya(i,j) or the like.

For example, the contour direction estimating unit 21 calculates an xdirection partial differential G_(x)(i,j) and a y direction partialdifferential G_(y)(i,j) of the signal value Y(i,j) of each pixel (i,j)using Formulae (2) and (3).

$\begin{matrix}\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 2} \rbrack & \; \\{{G_{x}( {i,j} )} = {\sum\limits_{u^{\prime},v^{\prime}}\;{{Y( {u,v} )}{W_{x}( {u^{\prime},v^{\prime}} )}}}} & (2) \\\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 3} \rbrack & \; \\{{G_{y}( {i,j} )} = {\sum\limits_{u^{\prime},v^{\prime}}\;{{Y( {u,v} )}{W_{y}( {u^{\prime},v^{\prime}} )}}}} & (3)\end{matrix}$

In Formulae (2) and (3), i and j are integer values indicating an indexof a pixel of interest in the x direction and the y direction,respectively. A pixel of interest is a pixel attracting attention as adirect processing target. W_(x)(u′,v′) and W_(y)(u′,v′) indicate filtercoefficients of the x direction and y direction differential filters,respectively. u and v are integer values indicating an index of areference pixel in the x direction and the y direction, respectively. Areference pixel is a pixel that is in a range decided according to apredetermined rule based on a pixel of interest and referred to whenprocessing on a pixel of interest is performed. A reference pixelincludes a pixel of interest. u′ and v′ are integer values indicating anindex of a reference pixel in the x direction and the y direction when apixel of interest is assumed as an original point, respectively. Thus,u=i+u′ and v=j+v′ are held.

For example, the differential filter has the filter coefficientsW_(x)(u′,v′) and W_(y)(u′,v′) for each of (u′,v′)s of a total of(2n+1)·(2n+1) reference pixels, that is, 2n+1 reference pixels in the xdirection and 2n+1 reference pixels in the y direction. In the followingdescription, a region to which the reference pixel given the filtercoefficient belongs is also referred to as a reference region. n is aninteger value (for example, 2) larger than 1. Here, the filtercoefficients W_(x)(u′,v′) and W_(y)(u′,v′) are 1 for a reference pixelin a positive direction based on a pixel of interest, are 0 for areference pixel having a coordinate value in the same differentialdirection (the x direction) as a pixel of interest, and are −1 for areference pixel in a negative direction based on a pixel of interest. Inother words, the filter coefficient W_(x)(u′,v′) of the x directiondifferential filter is 1 (0<u′ n), 0 (u′=0), or −1 (0>u′≧−n). The filtercoefficient W_(y)(u′,v′) of the y direction differential filter is 1(0<v′≦n), 0 (v′=0), or −1 (0>v′≧−n). Further, n is an integer value thatis equal to an enlargement factor of an image or larger than theenlargement factor. Thus, since the signal value is smoothed in thepositive direction and the negative direction based on the pixel ofinterest, noise generated by changing resolution can be reduced. Here,when n is large and a reference pixel away from a pixel of interest isconsidered, there are cases in which a partial differential valueserving as a local value originally is not properly calculated. Thus, nis decided to be a value smaller than a predetermined maximum value, forexample, an integer value equal to an enlargement factor, an integervalue obtained by rounding up a digit after a decimal point of anenlargement factor, or a value that is larger than any of the integervalues by a predetermined value.

The contour direction estimating unit 21 quantizes the contour directionθ(i,j) calculated based on the calculated x direction partialdifferential G_(x)(i,j) and the y direction partial differentialG_(y)(i,j), and calculates a quantization contour direction D(i,j)indicating the quantized contour direction. The contour directionestimating unit 21 calculates the quantization contour direction D(i,j),for example, using Formula (4).

$\begin{matrix}\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 4} \rbrack & \; \\{{D( {i,j} )} = {{round}( {\frac{N_{d}}{\pi}{\tan^{- 1}( {{G_{y}( {i,j} )}/{G_{x}( {i,j} )}} )}} )}} & (4)\end{matrix}$

In Formula (4), round ( . . . ) is a rounding function that provides aninteger value obtained by rounding off a digit after a decimal point ofa real number . . . . N_(d) is a constant indicating a number (aquantization contour direction number) of the quantized contourdirection. For example, the quantization contour direction number N_(d)is any of values of 8 to 32. Further, in order to avoid division byzero, when an absolute value |G_(x)(i,j)| of the x direction partialdifferential G_(x)(i,j) is smaller than a predetermined small realnumber value (for example, 10⁻⁶), π/2 is used as tan⁻¹. In other words,the quantization contour direction D(i,j) is indicated by any ofintegers of 0 to N_(d)−1 obtained by rounding a value obtained bydividing the contour direction θ by a quantization interval of π/N_(d).As a result, a degree of freedom of the contour direction θ isrestricted, and a processing load which will be described later isreduced.

The contour direction estimating unit 21 outputs quantization contourdirection information indicating the calculated quantization contourdirection D(i,j) to the direction evaluating unit 22 and the referenceregion weighting processing unit 23.

The direction evaluating unit 22 calculates a direction evaluation valueof each reference pixel belonging to a reference region centering on apixel of interest based on the quantization contour direction of eachpixel indicated by the quantization contour direction information inputfrom the contour direction estimating unit 21 for each pixel ofinterest. Here, the direction evaluating unit 22 decides the directionevaluation value of the reference pixel such that as the differencebetween the quantization contour direction D(i,j) of the pixel ofinterest (i,j) and the quantization contour direction D(u,v) of thereference pixel (u,v) decreases, the direction evaluation valueincreases. For example, the direction evaluating unit 22 calculates adifferential value D=D(u,v)−D(i,j) between the quantization contourdirection D(i,j) for the pixel of interest (i,j) and the quantizationcontour direction D(u,v) for the reference pixel (u,v). Here, when thedifferential value D is 0, that is, when D(u,v) is equal to D(i,j), adirection evaluation value F(|ΔD|) is decided to be a maximum value 1.When the differential value D is not 0, that is, when D(u,v) is notequal to D(i,j), the direction evaluation value F(|ΔD|) is decided to bea minimum value 0.

The direction evaluating unit 22 may decide the direction evaluationvalue F(ΔD) such that as the quantization contour direction D(i,j) forthe pixel of interest (i,j) approximates to the quantization contourdirection D(u,v) for the reference pixel (u,v), that is, as an absolutevalue |ΔD| of the differential value D decreases, the directionevaluation value F(ΔD) increases. For example, the direction evaluatingunit 22 decides F(0)=1, F(1)=0.75, F(2)=0.5, F(3)=0.25, andF(IADI)=0(|ΔD|>3).

When one of the quantization contour direction D(i,j) and thequantization contour direction D(u,v) is larger than N_(d)/2, and theother is smaller than N_(d)/2, since the absolute value |ΔD| increasesalthough the respective contour directions approximate to each other,there are cases in which an erroneous direction evaluation value F(ΔD)is calculated. For example, when D(i,j) is 7, and D(u,v) is 0, |ΔD| is7. However, the difference between the quantization contour directionsis π/8, and |ΔD| has to be decided to be 1 originally. In this regard,when one of the quantization contour direction D(i,j) and thequantization contour direction D(u,v) is larger than N_(d)/2, thedirection evaluating unit 22 adds N_(d) to a value of the otherquantization contour direction, and calculates a corrective value. Thedirection evaluating unit 22 calculates an absolute value of adifferential value between the calculated corrective value and onequantization contour direction. As a result, a desired directionevaluation value is decided using the calculated absolute value as |ΔD|described above.

In the product-sum operation unit 25 which will be described later, asthe direction evaluation value F(|ΔD|) is used, influence by thereference pixel (u,v) having the contour direction different from thecontour direction of the pixel of interest (i,j) can be ignored ordisregarded.

Even in the direction evaluating unit 22, the size of the referenceregion to which the reference pixel (u,v) belongs, that is, the numberof pixels in the horizontal direction or the vertical direction ispreferably 2n+1 or larger. Further, the size of the reference region inthe direction evaluating unit 22 may be different from the size of thereference region in the contour direction estimating unit 21. Forexample, the number of pixels of the reference region in the horizontaldirection and the vertical direction in the direction evaluating unit 22may be 7, respectively, whereas the number of pixels of the referenceregion in the horizontal direction and the vertical direction in thecontour direction estimating unit 21 may be 5, respectively.

The direction evaluating unit 22 outputs direction evaluation valueinformation indicating the direction evaluation value F(ΔD) of eachreference pixel (u,v) for each pixel of interest (i,j) to theproduct-sum operation unit 25. An exemplary numerical value of thedirection evaluation value F(ΔD) will be described later.

The reference region weighting processing unit 23 decides referenceregion weighting information for each pixel of interest (i,j) based onthe quantization contour direction D(i,j) of each pixel indicated by thequantization contour direction information input from the contourdirection estimating unit 21. The reference region weighting informationis information indicating a weighting coefficient R(D(i,j),u′,v′) ofeach reference pixel (u′,v′) belonging to a reference region centeringon a certain pixel of interest (i,j). This weighting coefficient is alsoreferred to as a reference region weighting. The size of the referenceregion in the reference region weighting processing unit 23 is decidedin advance to be equal to the size of the reference region in thedirection evaluating unit 22.

The reference region weighting processing unit 23 decides a value largerthan the weighting coefficients of the reference pixels in directions ofthe other ranges to be the weighting coefficient R(D(i,j),u′,v′) of thereference pixel in a direction of a predetermined range from thequantization contour direction D(i,j) of the pixel of interest (i,j).For example, the reference region weighting processing unit 23 decides“1” as the weighting coefficient R(D(i,j),u′,v′) of the reference pixel(u′,v′) in the quantization contour direction or a directionapproximating to the quantization contour direction from the pixel ofinterest (i,j), and decides “0” as the weighting coefficientsR(D(i,j),u′,v′) of the reference pixel (u′,v′) in the other directions.Specifically, the reference pixel in the quantization contour directionor a direction approximating to the quantization contour direction fromthe pixel of interest refers to the reference pixel (u′,v′) in which aline segment extending from the center of the pixel of interest (i,j) inthe quantization contour direction passes through the region. Thereference region weighting processing unit 23 may decide the weightingcoefficient such that the weighting coefficient has a large value forthe reference pixel (u′,v′) in which a length of the line segmentpassing through the region is large.

Further, the weighting coefficient of each reference pixel in eachquantization contour direction may be calculated in advance. Thereference region weighting processing unit 23 includes a storage unitthat stores in advance the reference region weighting informationindicating the calculated weighting coefficient of each reference pixelin association with the quantization contour direction information. Thereference region weighting processing unit 23 reads the reference regionweighting information corresponding to the quantization contourdirection indicated by the input quantization contour directioninformation from the storage unit.

The reference region weighting processing unit 23 outputs the referenceregion weighting information decided for each pixel of interest (i,j) tothe product-sum operation unit 25. An exemplary numerical value of thereference region weighting will be described later.

The pre-processing unit 24 extracts a brightness signal indicating thesignal value Y(u,v) of each reference pixel (u,v) belonging to thereference region centering on the pixel of interest (i,j) for each pixelof interest (i,j) from the brightness signal Y input from the scalingunit 13. The pre-processing unit 24 outputs the brightness signal Yextracted for each pixel of interest (i,j) to the product-sum operationunit 25. The size of the reference region in the pre-processing unit 24is decided in advance to be equal to the sizes of the reference regionsin the direction evaluating unit 22 and the reference region weightingprocessing unit 23.

The product-sum operation unit 25 receives the direction evaluationvalue information, the reference region weighting information, and thebrightness signal from the direction evaluating unit 22, the referenceregion weighting processing unit 23, and the pre-processing unit 24,respectively, for each pixel of interest (i,j).

The product-sum operation unit 25 calculates a product-sum value S(i,j),for example, using Formula (5) based on the direction evaluation valueF(ΔD) indicated by the direction evaluation value information, thereference region weighting R(D(i,j),u′,v′) indicated by the referenceregion weighting information, and the signal value Y(u,v) indicated bythe brightness signal.

$\begin{matrix}\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 5} \rbrack & \; \\{{S( {i,j} )} = {\sum\limits_{u^{\prime},v^{\prime}}\;{{F( {{\Delta\; D}} )}{R( {{D( {i,j} )},u^{\prime},v^{\prime}} )}{Y( {u,v} )}}}} & (5)\end{matrix}$

Formula (5) represents that the product of the direction evaluationvalue F(|ΔD|), the reference region weighting R(D(i,j),u′,v′), and thesignal value Y(u,v) indicated by the brightness signal is calculated foreach reference pixel, and the sum of the calculated products of thereference pixels belonging to the reference region is calculated as theproduct-sum value S(i,j). That is, in Formula (5), the product-sum valueS(i,j) is considered to be calculated by weighting and adding the signalvalue Y(u,v) using the product of the direction evaluation value F(|ΔD|)and the reference region weighting R(D(i,j),u′,v′) as the weightingcoefficient. The product of the direction evaluation value F(|ΔD|) andthe reference region weighting R(D(i,j),u′,v′) is also referred to as adirection evaluation region weighting.

The product-sum operation unit 25 calculates a weighting area C(i,j),for example, using Formula (6) based on the direction evaluation valueF(|ΔD|) indicated by the direction evaluation value information and thereference region weighting R(D(i,j),u′,v′) indicated by the referenceregion weighting information.

$\begin{matrix}\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 6} \rbrack & \; \\{{C( {i,j} )} = {\sum\limits_{u^{\prime},v^{\prime}}\;{{F( {{\Delta\; D}} )}{R( {{D( {i,j} )},u^{\prime},v^{\prime}} )}}}} & (6)\end{matrix}$

Formula (6) represents that the product of the direction evaluationvalue F(|ΔD|) and the reference region weighting R(D(i,j),u′,v′) iscalculated for each reference pixel, and the sum of the calculatedproducts of the reference pixels belonging to the reference region iscalculated as the weighting area C(i,j). That is, the weighting areaC(i,j) indicates a value obtained by weighting the reference regionweighting R(D(i,j),u′,v′) by the direction evaluation value F(|ΔD|) foreach reference pixel, that is, the number of reference pixels that isactually referred to in the product-sum operation of Formula (5). Inother words, Formula (6) represents the weighting area C(i,j) iscalculated by obtaining the sum of the direction evaluation regionweightings in the reference region. Further, the product-sum operationunit 25 calculates the sum of the reference region weightingsR(D(i,j),u′,v′) indicated by the reference region weighting informationfor the reference pixels belonging to the reference region as areference area N(i,j). The reference area N(i,j) indicates the number ofreference pixels that are nominally referred to in the product-sumoperation of Formula (5).

The product-sum operation unit 25 outputs product-sum value informationindicating the product-sum value S(i,j) calculated for each pixel ofinterest (i,j), weighting area information indicating the weighting areaC(i,j), and reference area information indicating the reference areaN(i,j) to the composition operation unit 26.

The composition operation unit 26 receives the product-sum valueinformation, the weighting area information, and the reference areainformation from the product-sum operation unit 25. Further, thecomposition operation unit 26 receives the parameter Pm from theparameter deciding unit 20 b. The composition operation unit 26calculates a direction smoothing value Y′(i,j) by dividing theproduct-sum value S(i,j) indicated by the product-sum value informationby the weighting area C(i,j) indicated by the weighting areainformation. That is, the calculated direction smoothing value Y′(i,j)indicates a signal value smoothed between the reference pixels that arein the quantization contour direction of the pixel of interest (i,j) orthe direction approximating to the quantization contour direction andhave the contour direction that is the same as or approximates to thecontour direction of the pixel of interest.

The composition operation unit 26 calculates a mixing ratio w(i,j) bydividing the weighting area C(i,j) by the reference area N(i,j)indicated by the reference area information. The mixing ratio w(i,j)indicates a ratio of the number of reference pixels having the contourdirection that is the same or approximates to the contour direction ofthe pixel of interest to the number of reference pixels that are in thequantization contour direction of the pixel of interest (i,j) or thedirection approximating to the quantization contour direction.

The composition operation unit 26 calculates a low pass signal valueY″(i,j) by performing a weighting addition (a composition operation) onthe direction smoothing value Y′(i,j) and the signal value Y(i,j)indicated by the brightness signal input from the scaling unit 13 usingw(i,j)Pm and (1−w(i,j)Pm), respectively. The weighting addition isrepresented by Formula (7).[Mathematical Formula 7]Y″(i,j)=w(i,j)Y′(i,j)+(1−w(i,j))Y(i,j)  (7)

The parameter Pm is a parameter of deciding a degree in which the effectof processing by the low pass filter unit 20 a is reflected as describedabove. For example, when Pm=0, Formula (7) becomes Y″(i,j)=Y(i,j), andthe effect of processing by the low pass filter unit 20 a is notreflected. On the other hand, when Pm=1,Y″(i,j)=w(i,j)Y′(i,j)+(1−w(i,j))Y(i,j), and the effect of processing bythe low pass filter unit 20 a is most strongly reflected.

The composition operation unit 26 generates the brightness signal Y″indicating the calculated low pass signal value Y″(i,j). The compositionoperation unit 26 outputs the generated brightness signal Y″ to the highfrequency expanding unit 27.

Next, a configuration of the high frequency expanding unit 27 will bedescribed.

FIG. 3 is a conceptual diagram illustrating a configuration of the highfrequency expanding unit 27 according to the present embodiment.

The high frequency expanding unit 27 is configured to include k (k is aninteger of 1 or larger) non-linear filter units 28-1 to 28-k and acomposition operation unit 29.

Each of the non-linear filter units 28-1 to 28-k calculates a frequencycomponent value for the low pass signal value Y″(i,j) of each pixelindicated by the brightness signal Y″ input from the compositionoperation unit 26, and outputs the calculated high frequency componentvalues NL₁ to NL_(k) to the composition operation unit 29.

Here, the non-linear filter units 28-1 to 28-k include linear filterunits 281-1 to 281-k and non-linear operation units 282-1 to 282-k,respectively.

Each of the linear filter units 281-1 to 281-k extracts a componentindicating a line drawing (for example, a contour) facing in a certaindirection. The linear filter units 281-1 to 281-k output a directioncomponent signal indicating the extracted component to the non-linearoperation units 282-1 to 282-k, respectively.

A specific example of the component extracted by the linear filter units281-1 to 281-k will be described later.

The non-linear operation units 282-1 to 282-k perform a non-linearoperation on signal values indicated by the direction component signalsinput from the linear filter units 281-1 to 281-k, and calculatenon-linear output values. The non-linear operation units 282-1 to 282-kgenerate direction component signals indicated by the calculatednon-linear output values, and output the generated direction componentsignals to the composition operation unit 29, respectively.

For example, the non-linear operation performed by the non-linearoperation units 282-1 to 282-k is a high dimensional function f(W) of aninput signal value W. For example, f(W) is sgn(W)|W|², W³, sgn(W)|W|⁴,W⁵ . . . , or a linear combination thereof. sgn( . . . ) indicates asignum function of a real number “ . . . .” In other words, sgn( . . . )is a function of outputting 1 when “ . . . ” is larger than 0, −1 when “. . . ” is smaller than 0, and 0 when “ . . . ” is 0. The function is anodd function, and thus an output value includes an odd-order harmoniccomponent.

The composition operation unit 29 receives high frequency componentvalues NL₁ to NL_(k) from the non-linear filter units 28-1 to 28-k.Further, the composition operation unit 29 receives the parameter Pnfrom the parameter deciding unit 20 b. The composition operation unitcalculates a high frequency extension signal value Z(i,j) by adding(synthesizing) the product of the high frequency component value NL₁ toNL_(k) and the parameter Pn and adding (synthesizing) the low passsignal value Y″(i,j) of the respective pixels. The composition operationunit 29 generates the brightness signal Z indicating the calculated highfrequency extension signal value Z(i,j). This addition is expressed asin Formula (8).

$\begin{matrix}\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 8} \rbrack & \; \\{{Z( {i,j} )} = {{Y^{''}( {i,j} )} + {{Pn}{\sum\limits_{k}\;{NL}_{k}}}}} & (8)\end{matrix}$

The parameter Pn is a parameter of deciding a degree in which the effectof processing by the high frequency expanding unit 27 is reflected asdescribed above. For example, when Pn=0, Formula (8) becomesZ(i,j)=Y″(i,j), and the effect of processing by the high frequencyexpanding unit 27 is not reflected. On the other hand, when Pn=1, Z(i,j)=Y″(i,j)+ΣNL_(k), and the effect of processing by the highfrequency expanding unit 27 is most strongly reflected.

Next, an exemplary configuration of the high frequency expanding unit 27will be described.

FIG. 4 is a schematic diagram illustrating an exemplary configuration ofthe high frequency expanding unit 27 according to the presentembodiment.

The high frequency expanding unit 27 according to the present exemplaryconfiguration generates harmonic components of components indicatingline drawings in the vertical direction and the horizontal direction.The high frequency expanding unit 27 is configured to include twonon-linear filter units 28-v and 28-h and a composition operation unit29. In other words, the non-linear filter units 28-v and 28-h areexamples of the non-linear filter units 28-1 and 28-2 (FIG. 3).

The non-linear filter unit 28-v generates a direction component signalindicating a line drawing in the vertical direction based on thebrightness signal Y″ input from the composition operation unit 26. Thenon-linear filter unit 28-v is configured to include a vertical highpass filter unit 281-v and a non-linear operation unit 282-v.

The vertical high pass filter unit 281-v and a horizontal high passfilter unit 281-h are examples of the linear filter unit 281-1 or thelike. The vertical high pass filter unit 281-v extracts a componentindicating a line drawing facing in the vertical direction, and outputsa vertical direction component signal W_(v) indicating the extractedcomponent to the non-linear operation unit 282-v. The horizontal highpass filter unit 281-h extracts a component indicating a line drawingfacing in the horizontal direction, and outputs a horizontal directioncomponent signal W_(h) indicating the extracted component to thenon-linear operation unit 282-h.

The non-linear operation units 282-v and 282-h are examples of thenon-linear operation unit 282-1 or the like. The non-linear operationunits 282-v and 282-h perform the above-described non-linear operationon signal values indicated by the direction component signals W_(v) andW_(h) input from the vertical high pass filter unit 281-v and thehorizontal high pass filter unit 281-h. The non-linear operation units282-v and 282-h generate a vertical direction high frequency componentvalue NL_(v) and a horizontal direction high frequency component valueNL_(h) indicated by the calculated non-linear output values, and outputthe generated vertical direction high frequency component value NL_(v)and the horizontal direction high frequency component value NL_(h) tothe composition operation unit 29. The composition operation unit 29 hasthe same configuration as the composition operation unit 29 describedabove.

Next, a configuration of the vertical high pass filter unit 281-v willbe described.

FIG. 5 is a schematic diagram illustrating a configuration of thevertical high pass filter unit 281-v according to the presentembodiment.

The vertical high pass filter unit 281-v is configured to include adelay memory 2811-v, a filter coefficient memory 2812-v, a multiplyingunit 2813-v, and a composition operation unit 2814-v.

The delay memories 2811-v output delay signals obtained by delaying alow-pass signal value based on the input brightness signal Y″ by W_(x),2·W_(x), . . . , (2n+1)·W_(x) samples to the multiplying unit 2813-v.W_(x) indicates the number of pixels in the horizontal directionincluded in an image of one frame. Thus, signal values of 2n+1 pixelsneighboring in the vertical direction centering on the pixel of interestare output to the multiplying unit 2813-v.

Here, the delay memory 2811-v includes 2n+1 delay elements 2811-v−1 to2811-v−2n+1 that delay an input signal by W_(x) samples. The delayelements 2811-v−1 to 2811-v−2n+1 are connected in series. One end of thedelay element 2811-v−1 receives the brightness signal Y″, and the otherend of the delay element 2811-v−1 outputs the delay signal obtained bydelaying by W_(x) samples to the multiplying unit 2813-v and one end ofthe delay element 2811-v−2. One ends of the delay element 2811-v−2 to2811-v−2n+1 receive the delay signals obtained by delaying by W_(x) to2n·W_(x) samples from the other ends of the delay elements 2811-v−1 to2811-v−2n. The other ends of the delay element 2811-v−2 to 2811-v−2noutput the delay signals obtained by delaying 2·W_(x) to 2n·W_(x)samples to one ends of the multiplying unit 2813-v and the delayelements 2811-v−3 to 2811-v−2n+1. The other end of the delay element2811-v−2n+1 outputs the delay signal obtained by delaying by(2n+1)·W_(x) samples to the multiplying unit 2813-v.

The filter coefficient memory 2812-v includes 2n+1 memory elements2812-v−1 to 2812-v−2n+1. The memory elements 2812-v−1 to 2812-v−2n+1store filter coefficients a_(L+1) to a_(L−n).

The multiplying unit 2813-v includes 2n+1 multipliers 2813-v−1 to2813-v−2n+1. The multipliers 2813-v−1 to 2813-v−2n+1 multiply the signalvalues input from the delay elements 2811-v−1 to 2811-v−2n+1 by thefilter coefficient a_(L+n) to a_(L−n) stored in the memory elements2812-v−1 to 2812-v−2n+1. The multipliers 2813-v−1 to 2813-v−2n+1 outputsmultiplication values obtained by the multiplying to the compositionoperation unit 2814-v.

The composition operation unit 2814-v calculates a composite value byadding the multiplication values input from the multipliers 2813-v−1 to2813-v−2n+1. The calculated composite value is a value obtained byperforming the product-sum operation on the signal values of 2n+1 pixelsneighboring the vertical direction centering on the pixel of interestand the filter coefficient a_(L+n) to a_(L−n). The composition operationunit 2814-v outputs the vertical direction component signal Wvindicating the calculated composite value to the non-linear operationunit 282-v.

Next, a configuration of the horizontal high pass filter unit 281-h willbe described.

FIG. 6 is a schematic diagram illustrating a configuration of thehorizontal high pass filter unit 281-h according to the presentembodiment.

The horizontal high pass filter unit 281-h is configured to include adelay memory 2811-h, a filter coefficient memory 2812-h, a multiplyingunit 2813-h, and a composition operation unit 2814-h.

The delay memory 2811-h, the filter coefficient memory 2812-h, themultiplying unit 2813-h, and the composition operation unit 2814-h havethe same configurations as the delay memory 2811-v, the filtercoefficient memory 2812-v, the multiplying unit 2813-v, and thecomposition operation unit 2814-v.

Here, the delay memory 2811-h includes 2n+1 delay elements 2811-h−1 to2811-h−2n+1 that delay an input signal by one sample instead of thedelay elements 2811-v−1 to 2811-v−2n+1 that delay an input signal by Wxsamples, respectively.

The filter coefficient memory 2812-h includes memory elements 2812-h−1to 2812-h−2n+1 instead of the memory elements 2812-v−1 to 2812-v−2n+1.The memory elements 2812-h−1 to 2812-h−2n+1 store filter coefficientsa_(D+n) to a_(D−n), respectively.

Thus, the composition operation unit 2814-h calculates a value obtainedby performing the product-sum operation on signal values of 2n+1 pixelsneighboring in the horizontal direction centering on the pixel ofinterest and the filter coefficients a_(D+n) to a_(D−n), respectively.The composition operation unit 2814-h outputs a horizontal directioncomponent signal W_(h) indicating the calculated composite value to thenon-linear operation unit 282-h.

The filter coefficients a_(L−n), a_(L−n+1), to a_(L+n) are high-passfilter coefficients used to implement a high pass filter through theproduct-sum operation with the signal value. A total value of the filtercoefficient a_(L+n) to a_(L−n) and a total value of the filtercoefficient a_(D+n) to a_(D−n) are 0, that is, a transfer function of adirect current (DC) component is 0. As a result, no DC component isincluded in the calculated composite value. The filter coefficienta_(L+n) to a_(L−n) have characteristics (high-pass characteristics) ofpassing through a frequency component higher than a spatial frequencythat is lower than a Nyquist frequency f_(nyq)′ of a non-enlargedbrightness signal by a predetermined frequency f. The Nyquist frequencyf_(nyq)′ of the non-enlarged brightness signal is a frequency obtainedby dividing a sampling frequency f_(s) of a brightness signal of aprocessing target by 2n (n is an enlargement factor). For example, thefilter coefficients a_(L+3), a_(L+2), a_(L+1), a_(L), a_(L−1), a_(L−2),and a_(L−3) are −0.0637, 0.0, 0.5732, −0.0189, 0.5732, 0.0000, and−0.0637. The filter coefficients a_(D+3), a_(D+2), a_(D+1), a_(D),a_(D−1), a_(D−2), and a_(D−3) may be values having high-passcharacteristics, similarly to the filter coefficients a_(L+3), a_(L+2),a_(L+1), a_(L), a_(L−1), a_(L−2), and a_(L−3), respectively.

Next, an exemplary configuration (a non-linear operation unit 282-A) ofthe non-linear operation unit 282-1 or the like will be described.

FIG. 7 is a schematic diagram illustrating the non-linear operation unit282-A according to the present embodiment.

The non-linear operation unit 282-A is configured to include an absolutevalue calculating unit 2821-A, an exponential operation unit 2822-A, afilter coefficient memory 2823-A, a multiplying unit 2824-A, acomposition operation unit 2825-A, a sign detecting unit 2826-A, and amultiplying unit 2827-A.

The non-linear operation unit 282-A outputs an l-order odd functionsgn|W|·(c₁·|W|+c₂·|W|²+ . . . +c_(l)·|W|^(l)) (l is an integer of 1 orlarger) on an input signal value W as a non-linear output value NL_(A).c₁, c₂, . . . , and c_(l) are 1, 2, . . . , and l-order coefficients.

The absolute value calculating unit 2821-A calculates the absolute value|W| of the signal value W indicated by the direction component signalinput from the linear filter unit 281-1 or the like, and outputs thecalculated absolute value |W| to the exponential operation unit 2822-A.

The exponential operation unit 2822-A includes (l−1) multipliers2822-A−2 to 2822-A−1, and outputs the absolute value |W| input from theabsolute value calculating unit 2821-A to the multiplying unit 2824-A.The multiplier 2822-A−2 calculates an absolute square value |W|² bymultiplying the absolute values |W| input from the absolute valuecalculating unit 2821-A. The multiplier 2822-A−2 outputs the calculatedabsolute square value |W|² to the multiplier 2822-A−3 and themultiplying unit 2824-A. The multipliers 2822-A−3 to 2822-A1−1 calculatethe absolute cube value |W|³ to an absolute (l−1)-th power value bymultiplying the absolute square value |W|² to an absolute (l−2)-th powervalue |W|¹⁻² input from the multipliers 2822-A−2 to 2822-A−l−2 by theabsolute value |W| input from the absolute value calculating unit2821-A. The multipliers 2822-A−3 to 2822-A−l−1 outputs the calculatedabsolute cube value |W|³ to the absolute (l−1)-th power value |W|¹⁻¹ tothe multiplier 2822-A−4 to 2822-A−1 and the multiplying unit 2824-A,respectively. The multiplier 2822-A−1 calculates an absolute 1-th powervalue |W|^(l) by multiplying the absolute (l−1)-th power value |W|^(l−1)input from the multiplier 2822-A−l−1 by the absolute value IWI inputfrom the absolute value calculating unit 2821-A. The multiplier 2822-A−1outputs the calculated absolute l-th power value |W|^(l) to themultiplying unit 2824-A.

The filter coefficient memory 2823-A includes 1 memory elements 2823-A−1to 2823-A−1. The memory elements 2823-A−1 to 2823-A−1 store the first tol-order coefficients c₁ to c_(l).

The multiplying unit 2824-A includes l multipliers 2824-A−1 to 2824-A−1.

The multipliers 2824-A−1 to 2824-A−1 calculate multiplication values bymultiplying the absolute value |W| to the absolute l-th power value|W|^(l) input from the exponential operation unit 2822-A by the first tol-order coefficients c₁ to c_(l) stored in the memory elements 2823-A−1to 2823-A−1. The multipliers 2824-A−1 to 2824-A−1 output the calculatedmultiplication values to the composition operation unit 2825-A.

The composition operation unit 2825-A calculates a composite value byadding the multiplication values input from the multipliers 2824-A−1 to2824-A−1. The composition operation unit 2825-A outputs the calculatedcomposite value to the multiplying unit 2827-A.

The sign detecting unit 2826-A detects a sign of the signal value Windicated by the direction component signal input from the linear filterunit 281-1 or the like, that is, a negative or positive sign of thesignal value W. When the signal value is smaller than 0, the signdetecting unit 2826-A outputs −1 to the multiplying unit 2827-A as asign value. When the signal value is 0 or larger, the sign detectingunit 2826-A outputs 1 to the multiplying unit 2827-A as a sign value.

The multiplying unit 2827-A calculates a high frequency component valueNL_(A) by multiplying the composite value input from the compositionoperation unit 2825-A by the sign value input from the sign detectingunit 2826-A. The multiplying unit 2827-A outputs the calculated highfrequency component value to the composition operation unit 29.

The non-linear operation unit 282-A having the above configuration has arelatively large circuit size, but can adjust the high frequencycomponent value to be output using a small number of coefficients.

Further, when the coefficient values c₁ to c_(l−1) excluding thehighest-order coefficient value, that is, the l-order coefficient valueare 0, the configuration related to the product-sum operation for suchorders may be omitted. A configuration that can be omitted includes thememory elements 2823-A−1 to 2823-A−l−1 and the multipliers 2824-A−1 to2824-A−l−1 in the non-linear operation unit 282-A. For example, whenf(W)=sgn(W)|W|², the memory element 2823-A−1 and the multiplier 2824-A−1may be omitted.

Next, another exemplary configuration (a non-linear operation unit282-B) of the non-linear operation unit 282-1 or the like will bedescribed.

FIG. 8 is a schematic diagram illustrating a configuration of thenon-linear operation unit 282-B according to the present embodiment.

The non-linear operation unit 282-B is configured to include a dataselecting unit 2828-B and a storage unit 2829-B.

The storage unit 2829-B stores an input value W and an output valueindicating a high dimensional function f(W) of the input value W inassociation with each other. In the example illustrated in FIG. 8, inputvalues 1.0, 4.0, . . . , and 16.0 and output values 1.0, 4.0, . . . ,and 256.0 are stored in association with each other.

The data selecting unit 2828-B searches for an input value equal to thesignal value W indicated by the direction component signal input fromthe linear filter unit 281-1 or the like in the storage unit 2829-B, andreads an output value corresponding to the searched input value. Thedata selecting unit 2828-B outputs the read output value to thecomposition operation unit 29 as the high frequency component valueNL_(B).

The non-linear operation unit 282-B having the above configuration hasmany input values to be set in advance and many output values, but has asmall circuit size. When the input value equal to the signal value W isnot stored in the data selecting unit 2828-B, the non-linear operationunit 282-B selects an input value that best approximates to the signalvalue W and is smaller or larger than the signal value W. The non-linearoperation unit 282-B may interpolate or extrapolate the output valuecorresponding to the selected input value and calculate the highfrequency component value NL_(B).

(Exemplary Contour)

Next, an exemplary contour will be described.

FIG. 9 is a conceptual diagram illustrating an exemplary contour.

In FIG. 9, a horizontal axis denotes the x direction, and a verticalaxis denotes the y direction. In an image 51, a magnitude of the signalvalue Y(x,y) is indicated by a contrasting density. A bright portion hasa large signal value Y(x,y), and a dark portion has a small signal valueY(x,y).

A dotted line passing through points of interest 52 and 53 indicates acontour 54. The point of interest 53 is at a position away from thepoint of interest 52 by a small amount (δx,δy). The contour 54 is a linesegment (a contour line) that passes through the point of interest 53and indicates positions at which the signal values are equal to thesignal value Y(x,y) of the point of interest 52. Generally, when thesignal value Y(x,y) is assumed to be differentiable for the coordinates(x,y), a difference Y between a signal value Y(x+δx,y+δy) of the pointof interest 53 and the signal value Y(x,y) of the point of interest 52is the sum of a small change δx·G_(x) (x,y) in the x direction and asmall change δy·G_(y) (x,y) in the y direction. Formula (1) is derivedfrom a relation in which the contour 54 passes through both of thepoints of interest 52 and 53, that is, Y is 0. Thus, the contourdirection estimating unit 21 can calculate the contour direction θ usingFormulae (1) to (3) based on the signal values Y(i,j) of the pixelswhose positions are spatially discrete.

(Exemplary Reference Region)

Next, an exemplary reference region will be described.

FIG. 10 is a conceptual diagram illustrating an exemplary referenceregion.

A relation between a horizontal axis and a vertical axis of FIG. 10 isthe same as in FIG. 9. An image 61 is an image indicated by thebrightness signal Y. Small quadrangles included in the image 61 indicatepixels. A numerical value shown in each quadrangle indicates the signalvalue Y(i,j) of each pixel. A contrasting density of each quadrangleindicates a magnitude of the signal value Y(i,j). A bright portion has alarge signal value Y(i,j), and a dark portion has a small signal valueY(i,j). In FIG. 10, the signal values Y(i,j) at the left side are largerthan those at the right side. Here, a boundary between a region having alarge signal value Y(i,j) and a region having a small signal valueY(i,j) is inclined to the left side as it is closer to both upper andlower ends and inclined to the right side as it is closer to the center.

A region surrounded by a dotted line at an upper side from the centralportion of the image 61 indicates a reference region 62. A quadrangle atthe center of the reference region 62 indicates a pixel of interest 63.Arrows that are indicated by i and j serving as a starting point anddirected toward the pixel of interest 63 indicate that an index of thepixel of interest 63 is (i,j). A quadrangle at a fourth column from aleftmost column of a bottom row of the reference region 62 indicates areference pixel 64. Arrows that are indicated by u and v serving as astarting point and directed toward the reference pixel 64 indicate thatan index of the reference pixel 64 is (u,v).

(Exemplary Differential Filter)

Next, an exemplary differential filter will be described.

FIG. 11 is a conceptual diagram illustrating an exemplary x directiondifferential filter (a differential filter 65).

A relation of a horizontal axis and a vertical axis in FIG. 11 is thesame as in FIG. 10.

Small quadrangles included in the differential filter 65 indicatereference pixels. The differential filter 65 is an x directiondifferential filter in which the number of pixels in the x direction andthe number of pixels in the y direction are 5, respectively. A numericalvalue shown in each quadrangle indicates the filter coefficientW_(x)(u′,v′). The filter coefficient W_(x)(u′,v′) is 1 when u′>0, 0 whenu′=0, and −1 when u′<0. A quadrangle at a fourth column from a leftmostcolumn of a bottom row of the differential filter 65 indicates areference pixel 66. Arrows that are indicated by u′ and v′ serving as astarting point and directed toward the reference pixel 66 indicate thatan index of the reference pixel 66 is (u′,v′), and corresponds to theindex (u,v) of the reference pixel 64 (FIG. 10). Thus, in theproduct-sum operation of Formula (2), a signal value 93 related to thereference pixel 64 is multiplied by the filter coefficient 1 related tothe reference pixel 66.

FIG. 12 is a conceptual diagram illustrating an exemplary y directiondifferential filter (a differential filter 67).

A relation of a horizontal axis and a vertical axis in FIG. 12 is thesame as in FIG. 10.

Small quadrangles included in the differential filter 67 indicatereference pixels. The differential filter 67 is a vertical directiondifferential filter in which the number of pixels in the x direction andthe number of pixels in the y direction are 5, respectively. A numericalvalue shown in each quadrangle indicates the filter coefficientW_(y)(u′,v′). The filter coefficient W_(y)(u′,v′) is 1 when v′>0, 0 whenv′=0, and −1 when v′<0. A quadrangle at a fourth column from a leftmostcolumn of a bottom row of the differential filter 67 indicates areference pixel 68. Arrows that are indicated by u′ and v′ serving as astarting point and directed toward the reference pixel 68 indicate thatan index of the reference pixel 68 is (u′,v′), and corresponds to theindex (u,v) of the reference pixel 64 (FIG. 10). Thus, in theproduct-sum operation of Formula (3), the signal value 93 related to thereference pixel 64 is multiplied by the filter coefficient 1 related tothe reference pixel 68.

(Exemplary Calculation of Partial Differential)

FIG. 13 is a conceptual diagram illustrating an exemplary x directionpartial differential (an x direction partial differential 69).

A relation among a horizontal axis, a vertical axis, and the pixel ofinterest 63 in FIG. 13 is the same as in FIG. 10.

Small quadrangles included in the x direction partial differential 69indicate pixel. A numerical value shown in each quadrangle indicates avalue of the x direction partial differential G_(x)(i,j). The xdirection partial differential G_(x)(i,j) illustrated in FIG. 13 is avalue calculated using Formula (2) based on the signal value Y(i,j)illustrated in FIG. 10. A contrasting density of each quadrangleindicates a magnitude of the x direction partial differentialG_(x)(i,j). A bright portion has a large x direction partialdifferential G_(x)(i,j), and a dark portion has a small x directionpartial differential G_(x)(i,j). In FIG. 13, the x direction partialdifferentials G_(x)(i,j) at both left and right ends are small. Here,regions having a large x direction partial differential G_(x)(i,j) areinclined to the left side as it is closer to both upper and lower endsand inclined to the right side as it is closer to the center. Thiscorresponds to what the boundary between the region having the largesignal value Y(i,j) and the regions having the small signal value Y(i,j)is inclined to the left side as it is closer to both upper and lowerends and inclined to the right side as it is closer to the center inFIG. 10. The x direction partial differential G_(x)(i,j) of the pixel ofinterest 63 is 121.

FIG. 14 is a conceptual diagram illustrating an exemplary y directionpartial differential (a y direction partial differential 70).

A relation among a horizontal axis, a vertical axis, and the pixel ofinterest 63 in FIG. 14 is the same as in FIG. 10.

Small quadrangles included in the y direction partial differential 70indicate pixels. A numerical value shown in each quadrangle indicate avalue of the y direction partial differential G_(y)(i,j). The ydirection partial differential G_(y)(i,j) illustrated in FIG. 14 is avalue calculated using Formula (3) based on the signal value Y(i,j)illustrated in FIG. 10. A contrasting density of each quadrangleindicates a magnitude of the y direction partial differentialG_(y)(i,j). A bright portion has a large y direction partialdifferential G_(y)(i,j), and a dark portion has a small y directionpartial differential G_(y)(i,j). In FIG. 14, the y direction partialdifferential G_(x)(i,j) approximates to an intermediate value 0 at bothleft and right ends. Here, regions having the large y direction partialdifferential G_(x)(i,j) are inclined to the left side as it is closer tothe lower end and inclined to the right side as it is closer to thecenter. Regions having the small y direction partial differentialG_(x)(i,j) are inclined to the left side as it is closer to the upperend and inclined to the right side as it is closer to the center. Theregions having the large y direction partial differential G_(y)(i,j) andthe regions having the small y direction partial differential G_(y)(i,j)are almost symmetric centering on the central axis in the y direction.

This corresponds to what the boundary between the regions having thelarge signal value Y(i,j) and the regions having the small signal valueY(i,j) is inclined to the left side as it is closer to both upper andlower ends and inclined to the right side as it is closer to the centerin FIG. 14. The y direction partial differential G_(y)(i,j) of the pixelof interest 63 is −103.

(Exemplary Contour Direction Calculation)

Next, an exemplary contour direction calculation will be described.

FIG. 15 illustrates exemplary quantization contour direction candidates.

In FIG. 15, a horizontal axis denotes the x direction, and a verticalaxis denotes the y direction. In this example, the quantization contourdirection number N_(d) is 8. Arrows radially extending from one originalpoint indicate quantization contour direction candidates, and a numberat the end point of each arrow is a numerical value indicating thequantization contour direction. In other words, numerical values 0 to 7indicate contour direction angles 0 to 7π/8 or π to 15π/8. A regionsurrounded by a dotted line extending from an original point centeringon each arrow indicates a range of the contour direction (beforequantization) to be quantized in the quantization contour directionindicated by each arrow. For example, when the contour direction angleis included in any of 0 to π/16, 15π/16 to 17π/16, and 31π/16 to 2π, thenumerical value indicating the quantization contour direction is 0.

FIG. 16 is a conceptual diagram illustrating an exemplary quantizationcontour direction calculation (a quantization contour direction 71).

A relation among a horizontal axis, a vertical axis, and the pixel ofinterest 63 in FIG. 16 is the same as in FIG. 10.

Small quadrangles included in the quantization contour direction 71indicate pixels. Arrows that are indicated by i₂ and j₂ serving as astarting point and directed toward a pixel of interest 72 positioned ina left lower portion of the quantization contour direction 71 representthat an index of the pixel of interest 72 is (i₂,j₂).

In FIG. 16, reference regions 73 and 75 are regions in which the numberof pixels in the x direction centering on the pixels of interest 63 and72 is 7, and the number of pixels in the y direction is 7. Arrows thatare indicated by u and v serving as a starting point and directed towarda reference pixel 74 positioned in a right lower portion of the pixel ofinterest 63 represent that an index of the reference pixel 74 is (u,v).Arrows that are indicated by u₂ and v₂ serving as a starting point anddirected toward a reference pixel 76 positioned in a left lower portionof the pixel of interest 72 represent that an index of the referencepixel 76 is (u₂,v₂). The reference regions 73 and 75 and the referencepixels 74 and 76 will be described later.

A numerical value and an arrow shown in each quadrangle indicate a valueof the quantization contour direction D(i,j) and a contour directionindicated by the value. The quantization contour direction D(i,j)illustrated in FIG. 16 is a value that is calculated using Formula (4)based on the x direction partial differential G_(x)(i,j) illustrated inFIG. 13 and the y direction partial differential G_(y)(i,j) illustratedin FIG. 14. As a result, the quantization contour direction in most ofthe pixels at the left upper side from the center of FIG. 16 is a leftupper direction. The quantization contour direction in most of thepixels at the left lower side from the center is a right upperdirection. This corresponding to what the boundary between the regionshaving the large signal value Y(i,j) and the regions having the smallsignal value Y(i,j) is inclined to the left side as it is closer to bothupper and lower ends and inclined to the right side as it is closer tothe center.

Here, the quantization contour direction D(i,j) in the pixel of interest63 is 6. Thus, the reference region weighting processing unit 23 selectsthe reference region weighting R(6,u′,v′) for the pixel of interest 63.The quantization contour direction D(i₂,j₂) in the pixel of interest 72is 2. Thus, the reference region weighting processing unit 23 selectsthe reference region weighting R(2,u′,v′) for the pixel of interest 72.

(Exemplary Reference Region Weighting)

Next, an exemplary reference region weighting will be described.

FIG. 17 is a conceptual diagram illustrating an exemplary referenceregion weighting.

A relation between a horizontal axis and a vertical axis in FIG. 17 isthe same as in FIGS. 15 and 16.

Arrows radially extending from one original point at the lower side thanthe center of FIG. 17 and numerical values shown at end points of thearrows indicate quantization contour directions, similarly to FIG. 15.Each of grid-like diagrams in directions in which arrows indicated bynumerical values 0 to 7 counterclockwise from the right side of anoriginal point move toward an upper half plane indicates the referenceregion weighting R(D(i,j),u′,v′) corresponding to the quantizationcontour direction D(i,j). FIG. 17 illustrates the reference regionweightings R(0,u′,v′) to R (7,u′,v′) in the counterclockwise order fromthe right side. Small quadrangles included in each reference regionweighting R(D(i,j),u′,v′) indicate reference images. In FIG. 17, thenumber of reference images included in each reference region weightingR(D(i,j),u′,v′) is 7 in the horizontal direction and 7 in the verticaldirection. A numerical value shown in each quadrangle is a referenceregion weighting value. The reference region weighting value is 1 forthe reference pixel in the quantization contour direction from thecenter (the pixel of interest) of the reference region, and has a valuethat increases as the direction of the reference pixel approximates moreto the direction. For example, the reference region weighting R(0,u′,v′)corresponding to the quantization contour direction 0 is 1 for all thereference pixels in a fourth row, and 0 for all the reference pixels inthe other rows. The reference region weighting R(6,u′,v′) correspondingto the quantization contour direction 6 is 1 for all the referencepixels in a first row of a first column to a seventh row of a seventhcolumn and 0 for all the other reference pixels. The quantizationcontour direction is not necessarily the horizontal direction, thevertical direction, or a diagonal direction (π/4 or 3π/4). When thequantization contour direction is any other direction, the referenceregion weighting R(D(i,j),u′,v′) is decided to be proportional to adistance by which the line segment extending from the center of thereference region in the quantization contour direction passes through.For example, the reference region weighting R(1,u′,v′) corresponding tothe quantization contour direction 1 for a pixel in a seventh columnfrom a leftmost column of a second row from a topmost row is 0.2. Thereference region weightings R(1,u′,v′) corresponding to the quantizationcontour direction 1 for pixels in fifth to seventh columns of a thirdrow are 0.4, 0.8, and 0.8, respectively. The reference region weightingsR(1,u′,v′) corresponding to the quantization contour direction 1 forpixels in second to sixth columns of a fourth row are 0.2, 0.6, 1.0,0.6, and 0.2, respectively. The reference region weighting R(1,u′,v′)corresponding to the quantization contour direction 1 for pixels infirst to third columns of a fifth row are 0.8, 0.8, and 0.4,respectively. The reference region weighting R(1,u′,v′) corresponding tothe quantization contour direction 1 for a pixel in a first column of asixth row is 0.2. The reference region weightings R(1,u′,v′)corresponding to the quantization contour direction 1 for the otherreference pixels are 0.

FIG. 18 is a conceptual diagram illustrating another exemplary referenceregion weighting.

A relation between a horizontal axis and a vertical axis of and arelation between arrows indicating the quantization contour directionsand grid-like diagrams corresponding to the quantization contourdirections in FIG. 18 are the same as in FIG. 17. The quantizationcontour directions for the reference region weightings R(D(i,j),u′,v′)corresponding to the quantization contour directions 0, 2, 4, and 6 thatare the horizontal direction, the vertical direction, and the diagonaldirection (π/4 and 3π/4) are the same as in the example illustrated inFIG. 17.

The reference region weightings R(D(i,j),u′,v′) for the otherquantization contour directions are different from those of FIG. 17. Inthe example illustrated in FIG. 18, the reference region weightingsR(D(i,j),u′,v′) for the reference pixels through which a line segmentextending from the center of the reference region in the quantizationcontour direction passes are decided to be 1. The reference regionweighting R(D(i,j),u′,v′) for the other reference pixels are decided tobe 1. For example, the reference region weightings R(1,u′,v′)corresponding to the quantization contour direction 1 for pixels in aseventh column from a leftmost column in a second row from a topmostrow, pixels in fifth to seventh columns of a third row, pixels in secondto sixth columns of a fourth row, pixels in first to third columns of afifth row, and pixels in a first column of a sixth row are 1. Thereference region weightings R(1,u′,v′) corresponding to the quantizationcontour direction 1 for the other reference pixels are 0. In otherwords, in the example illustrated in FIG. 18, the reference regionweighting R(D(i,j),u′,v′) indicates whether or not the reference pixelis selected according to whether or not the line segment extending fromthe center of the reference region in the quantization contour directionpasses through.

(Exemplary Direction Evaluation Value Calculation)

Next, an exemplary direction evaluation value calculation will bedescribed.

FIG. 19 is a conceptual diagram illustrating an exemplary directionevaluation value (a direction evaluation value 77).

A horizontal axis and a vertical axis of FIG. 19 are the same as in FIG.11.

Small quadrangles included in the direction evaluation value 77 indicatereference pixels. A numerical value shown in each quadrangle indicates avalue of the direction evaluation value F(ΔD) for the pixel of interest63 (see FIG. 16). The direction evaluation value F(ΔD) is the directionevaluation value that is calculated based on D(i,j) for the pixel ofinterest 63 (see FIG. 16) and the quantization contour direction D(u,v)for each reference pixel (u,v) belonging to the reference region 73 (seeFIG. 16) through the direction evaluating unit 22. In FIG. 19, thedirection evaluation values F(ΔD) for the reference pixels in a topmostrow to a fourth row are 1. The direction evaluation values F(ΔD) for thereference pixels in a fifth row to a seventh row excluding the referencepixels in a leftmost fifth row are 0. In other words, FIG. 19illustrates that the quantization contour directions related to thereference pixels in the topmost row to the fourth row of the referenceregion 73 (see FIG. 16) are the same to the quantization contourdirections related to the pixel of interest 63 (see FIG. 16). FIG. 19illustrates that the quantization contour directions related to most ofthe reference pixels in the fifth to seventh rows of the referenceregion 73 (see FIG. 16) are different from the quantization contourdirections related to the pixel of interest 63 (see FIG. 16).

Here, arrows that are indicated by i and j serving as a starting pointand directed toward a reference pixel 78 indicate that the referencepixel 78 is a reference pixel corresponding to the pixel of interest 63having the same index (i,j). It indicates that the direction evaluatingunit 22 has decided “1” as the direction evaluation value F(ΔD) becausethe quantization contour direction D(i,j) of the pixel of interest 63 isthe same as the quantization contour direction D(i,j) of the pixel ofinterest 63 serving as the reference pixel.

Arrows that are indicated by u and v serving as a starting point anddirected toward a reference pixel 79 indicate that the reference pixel79 is a reference pixel corresponding to the reference pixel 74 havingthe same index (u,v). It indicates that the direction evaluating unit 22has decided “0” as the direction evaluation value F(ΔD) because thequantization contour direction D(i,j) of the pixel of interest 63 andthe quantization contour direction D(i,j) of the reference pixel 74 are5 and 4, respectively, and different from each other.

FIG. 20 is a conceptual diagram illustrating another exemplary directionevaluation value (a direction evaluation value 80).

A horizontal axis and a vertical axis of FIG. 20 are the same as in FIG.12.

Small quadrangles included in the direction evaluation value 80 indicatereference pixels. A numerical value shown in each quadrangle indicates avalue of the direction evaluation value F(ΔD) for the pixel of interest72 (see FIG. 16). The direction evaluation value F(ΔD) is the directionevaluation value that is calculated based on D(i₂,j₂) for the pixel ofinterest 72 (see FIG. 16) and the quantization contour directionD(u₂,v₂) for each reference pixel (u₂,v₂) belonging to the referenceregion 75 (see FIG. 16) through the direction evaluating unit 22. InFIG. 20, the direction evaluation values F(ΔD) for all the referencepixels are 1. In other words, FIG. 20 illustrates that the quantizationcontour directions related to all the reference pixels belonging to thereference region 75 (see FIG. 16) are the same as the quantizationcontour directions related to the pixel of interest 72 (see FIG. 16).

Here, arrows that are indicated by i₂ and j₂ serving as a starting pointand directed toward a reference pixel 81 indicate that the referencepixel 81 is a reference pixel corresponding to the pixel of interest 72(see FIG. 16) having the same index (i₂,j₂). It indicates that thedirection evaluating unit 22 has decided “1” as the direction evaluationvalue F(ΔD) because the quantization contour direction D(i₂,j₂) of thepixel of interest 72 is the same as the quantization contour directionD(i₂,j₂) of the pixel of interest 72 (see FIG. 16) serving as thereference pixel.

Arrows that are indicated by u₂ and v₂ serving as a starting point anddirected toward a reference pixel 82 indicate that the reference pixel82 is a reference pixel corresponding to the reference pixel 76 (seeFIG. 16) having the same index (u₂,v₂). It indicates that the directionevaluating unit 22 has decided “1” as the direction evaluation valueF(ΔD) because the quantization contour direction D(i₂,j₂) of the pixelof interest 72 (see FIG. 16) and the quantization contour directionD(i₂,j₂) of the reference pixel 76 (see FIG. 16) are 2 and 2,respectively, and the same as each other.

(Exemplary Direction Evaluation Region Weighting Calculation)

Next, an exemplary direction evaluation region weighting calculationwill be described.

FIG. 21 is a conceptual diagram illustrating an exemplary directionevaluation region (the direction evaluation region weighting 83).

A horizontal axis and a vertical axis of FIG. 21 are the same as in FIG.11.

Small quadrangles included in the direction evaluation region weighting83 indicate reference pixels. A numerical value shown in each quadrangleindicates a direction evaluation region weighting value corresponding tothe reference pixel. The direction evaluation region weighting value isa value that is obtained by multiplying the reference region weightingR(6,u′,v′) (see FIG. 17) of each reference pixel for the pixel ofinterest 63 (see FIG. 16) by the direction evaluation value F(ΔD) of thecorresponding reference pixel (see FIG. 19) through the product-sumoperation unit 25. In FIG. 21, the direction evaluation region weightingvalues for a reference pixel in a topmost leftmost column to a referencepixel (the reference pixels 78) in a fourth column of a fourth roware 1. The direction evaluation region weighting values for the otherreference pixels are 0.

Here, arrows that are indicated by i and j serving as a starting pointand directed toward a reference pixel 78 indicate that the referencepixel 78 corresponds to the reference pixel 78 (see FIG. 19) having thesame index (i,j). The reference pixel 78 corresponds to the pixel ofinterest 63 (see FIG. 16). In other words, it indicates that thedirection evaluation region weighting value for the reference pixel 78is calculated by multiplying the direction evaluation value F(ΔD) (seeFIG. 19) for the reference pixel 78 by the reference region weightingR(6,0,0) (see FIG. 17) corresponding to the reference pixel. Arrows thatare indicated by u and v serving as a starting point and directed towarda reference pixel 79 correspond to the reference pixel 79 having thesame index (u,v). The reference pixel 79 corresponds to the referencepixel 74 (see FIG. 19). In other words, it indicates that the directionevaluation region weighting value for the reference pixel 79 iscalculated by multiplying the direction evaluation value F(ΔD) (see FIG.19) for the reference pixel 79 by the reference region weightingR(6,2,2) (see FIG. 17) corresponding to the reference pixel.

FIG. 22 is a conceptual diagram illustrating another exemplary directionevaluation region weighting (a direction evaluation region weighting84).

A horizontal axis and a vertical axis of FIG. 22 are the same as in FIG.12.

Small quadrangles included in the direction evaluation region weighting84 indicate reference pixels. A numerical value shown in each quadrangleindicates a direction evaluation region weighting value corresponding tothe reference pixel. The direction evaluation region weighting value isa value that is obtained by multiplying the reference region weightingR(2,u′,v′) (see FIG. 17) of each reference pixel for the pixel ofinterest 72 (see FIG. 16) by the direction evaluation value F(ΔD) of thecorresponding reference pixel (see FIG. 19) through the product-sumoperation unit 25. In FIG. 22, the direction evaluation region weightingvalues for all the reference pixels on a diagonal line from a leftmostcolumn of a lowest row to a rightmost column of a topmost row are 1. Thedirection evaluation region weighting values for the other referencepixels are 0.

Here, arrows that are indicated by i₂ and j₂ serving as a starting pointand directed toward the reference pixel 81 indicate that the referencepixel 81 corresponds to the reference pixel 81 (see FIG. 22) having thesame index (i₂,j₂). The reference pixel 81 corresponds to the pixel ofinterest 72 (see FIG. 16). In other words, it indicates that thedirection evaluation region weighting value for the reference pixel 81is calculated by multiplying the direction evaluation value F(ΔD) (seeFIG. 20) for the reference pixel 81 by the reference region weightingR(2,0,0) (see FIG. 17) corresponding to the reference pixel. Arrows thatare indicated by u₂ and v₂ serving as a starting point and directedtoward the reference pixel 79 correspond to a reference pixel 82 havingthe same index (u₂,v₂). The reference pixel 76 (see FIG. 16) correspondsto the reference pixel 82. In other words, it indicates that thedirection evaluation region weighting value for the reference pixel 82is calculated by multiplying the direction evaluation value F(ΔD) (seeFIG. 20) for the reference pixel 82 by the reference region weightingR(2,−2,2) (see FIG. 17) corresponding to the reference pixel.

Next, a pixel (i,j) related to the signal value Y(i,j) that ismultiplied by the direction evaluation region weighting value having avalue (for example, 1) other than 0 in the product-sum operation unit 25will be described. Hereinafter, this pixel is referred to as a smoothingtarget pixel.

FIG. 23 is a conceptual diagram illustrating an exemplary smoothingtarget pixel.

A relation between a horizontal axis and a vertical axis of FIG. 23 isthe same as in FIG. 10. An image 61 is the same as the image 61illustrated in FIG. 10. In FIG. 23, a positional relation between pixelsof interest 63 and 72 and reference regions 73 and 75 is the same as thepositional relation in FIG. 10. In other words, the direction evaluationregion weightings related to the reference regions 73 and 75 are thesame as the direction evaluation region weighting 83 (see FIG. 21) andthe direction evaluation region weighting 84 (see FIG. 22),respectively. The direction evaluation region weightings 83 and 85 areused when the product-sum operation unit 25 calculates the product-sumvalue S(i,j), the weighting area C(i,j), and the reference area N(i,j)corresponding to the pixels of interest 63 and 72.

Small quadrangles drawn by thick lines in the reference regions 73 and75 indicate smoothing target pixels. The smoothing target pixelsincluded in the reference region 73 are reference pixels in a topmostleftmost column to a fourth column of a fourth row. The smoothing targetpixels included in the reference region 75 are all reference pixels on adiagonal line from a leftmost column of a lowest row to a rightmostcolumn of a topmost row. Thus, the signal values Y(i,j) related to thereference pixels are actually used to calculate the direction smoothingvalues Y′(i,j) related to the pixels of interest 63 and 72. On the otherhand, in the reference region 73, the reference pixels in a fifth columnof a fifth row to a lowest column of a topmost row are not actually usedto calculate the direction smoothing value Y′(i,j).

Thus, in the present embodiment, the contour direction of each pixel iscalculated, and the pixel of interest is smoothed using the signal valuerelated to the reference pixel that is in the quantization contourdirection of the pixel of interest or in the direction approximating tothe quantization contour direction and has the contour direction that isthe same as or approximates to the contour direction of the pixel ofinterest. Thus, it is possible to visually reduce noise while carefullyconsidering a reference pixel serving as an actual smoothing target evenat a corner point serving as a pixel that is different in the contourdirection from a neighboring or adjacent pixel.

Next, image processing according to the present embodiment will bedescribed.

FIG. 24 is a flowchart illustrating image processing according to thepresent embodiment.

(Step S100) The parameter deciding unit 20 b decides the parameter Pmand the parameter Pn based on the signal Ss input from the input unit11. The parameter deciding unit 20 b outputs the parameter Pm to the lowpass filter unit 20 a, and outputs the parameter Pn to the highfrequency expanding unit 27. Thereafter, the process proceeds to stepS101.

(Step S101) The contour direction estimating unit 21 calculates thecontour direction of each pixel based on the signal value of each pixelindicated by the brightness signal Y input from the scaling unit 13. Thecontour direction estimating unit 21 quantizes the contour direction ofeach calculated pixel, and calculates the quantization contourdirection. The contour direction estimating unit 21 outputs thequantization contour direction information indicating the calculatedquantization contour direction to the direction evaluating unit 22 andthe reference region weighting processing unit 23. Thereafter, theprocess proceeds to step S102.

(Step S102) The direction evaluating unit 22 calculates the directionevaluation value of each reference pixel belonging to the referenceregion centering on the pixel of interest for each pixel of interestbased on the quantization contour direction of each pixel indicated bythe quantization contour direction information input from the contourdirection estimating unit 21. For example, the direction evaluating unit22 calculates the direction evaluation value such that the referencepixel having the quantization contour direction that is the same orapproximates to the quantization contour direction of the pixel ofinterest has a large direction evaluation value. The directionevaluating unit 22 outputs the direction evaluation value informationindicating the direction evaluation value of each reference pixel foreach pixel of interest to the product-sum operation unit 25. Thereafter,the process proceeds to step S103.

(Step S103) The reference region weighting processing unit 23 decidesthe reference region weighting information for each the pixel ofinterest based on the quantization contour direction information inputfrom the contour direction estimating unit 21. The reference regionweighting processing unit 23 reads the reference region weightinginformation corresponding to the quantization contour direction of eachpixel of interest from the storage unit. The reference region weightinginformation indicates a weighting coefficient that increases as thereference pixel is in the quantization contour direction of the pixel ofinterest or the direction approximating to the quantization contourdirection. The reference region weighting processing unit 23 outputs theread reference region weighting information to the product-sum operationunit 25. Thereafter, the process proceeds to step S104.

(Step S104) The pre-processing unit 24 extracts the brightness signalindicating the signal value Y(u,v) of each reference pixel (u,v)belonging to the reference region centering on the pixel of interest(i,j) from the brightness signal Y input from the scaling unit 13 foreach pixel of interest (i,j) (pre-processing). The pre-processing unit24 outputs the brightness signal Y extracted for each pixel of interest(i,j) to the product-sum operation unit 25. Thereafter, the processproceeds to step S105.

(Step S105) The product-sum operation unit 25 receives the directionevaluation value information, the reference region weightinginformation, and the brightness signal from the direction evaluatingunit 22, the reference region weighting processing unit 23, and thepre-processing unit 24, respectively, for each pixel of interest. Theproduct-sum operation unit 25 calculates a product-sum value, forexample, using Formula (5) based on the direction evaluation valueindicated by the direction evaluation value information, the referenceregion weighting indicated by the reference region weightinginformation, and the signal value indicated by the brightness signal.The product-sum operation unit 25 calculates the weighting area, forexample, using Formula (6) based on the direction evaluation valueindicated by the direction evaluation value information, and thereference region weighting indicated by the reference region weightinginformation. The product-sum operation unit 25 calculates the sum of thereference region weightings indicated by the reference region weightinginformation for the reference pixels belonging to the reference regionas the reference area. The product-sum operation unit 25 generates theproduct-sum value information and the reference area informationrespectively indicated by the product-sum value, the weighting area, andthe reference area, and outputs the generated product-sum valueinformation, the weighting area information, and the reference areainformation to the composition operation unit 26. Thereafter, theprocess proceeds to step S106.

(Step S106) The composition operation unit 26 receives the parameter Pmfrom the parameter deciding unit 20 b, receives the product-sum valueinformation, the weighting area information and the reference areainformation from the product-sum operation unit 25, and receives thebrightness signal from the scaling unit 13. The composition operationunit 26 calculates the direction smoothing value by dividing theproduct-sum value indicated by the product-sum value information by theweighting area C indicated by the weighting area information. Thecomposition operation unit 26 calculates the mixing ratio by dividingthe weighting area by the reference area indicated by the reference areainformation. The composition operation unit 26 performs a compositionoperation by weighting and adding the direction smoothing value and thesignal value indicated by the brightness signal based on the mixingratio and the parameter Pm, and calculates a composition signal value.

The composition operation unit 26 outputs the brightness signalindicated by the calculated composition signal value to the highfrequency expanding unit 27. Thereafter, the process proceeds to stepS107.

(Step S107) The high frequency expanding unit 27 calculates the highfrequency component value based on the signal value indicated by thebrightness signal input from the composition operation unit 26. The highfrequency expanding unit 27 calculates the high frequency extensionsignal value by weighting-adding the calculated high frequency componentvalue and the signal value indicated by the input brightness signalbased on the parameter Pn. The high frequency expanding unit 27 outputsthe brightness signal indicating the calculated high frequency extensionsignal value to the image format converting unit 14. Thereafter, theprocess ends.

As described above, in the present embodiment, the contour direction ofeach pixel is estimated based on the signal value of each pixel ofinterest. In the present embodiment, the signal value of each pixel ofinterest is filtered using the signal value of each reference pixel thatis arranged in the estimated contour direction of each pixel of interestand in a predetermined reference region from each pixel of interest.Further, in the present embodiment, the high frequency component of thefiltered signal value of each pixel of interest is generated, and thefrequency band of the signal value of each pixel of interest isexpanded.

Thus, the signal value of each pixel of interest is smoothed in thecontour direction, and the noise in the visually sensitive contourdirection is removed or reduced. Further, since the high frequencycomponent of the smoothed signal value is compensated, it is possible tosharpen an image while preventing or suppressing noise caused byaliasing.

Further, in the present embodiment, intensity of smoothing and highfrequency extension is controlled according to an enlargement factor ofan image. Accordingly, it is possible to prevent the occurrence of anartifact when the enlargement factor is small.

Second Embodiment

FIG. 25 is a schematic diagram illustrating a configuration of a displaydevice 2 according to a second embodiment of the present invention.

The display device 2 differs from the display device 1 in a method ofdeciding the parameter Pm and the parameter Pn. In the display device 2,the scaling unit 13 outputs resolution ratio St serving as a ratio ofthe resolution of an input image and the resolution of an output imageto the image processing unit 20. More specifically, the scaling unit 13estimates horizontal resolution and vertical resolution from ahorizontal synchronous signal and a vertical synchronous signal includedin an input signal, respectively, and calculates the resolution ratio Stof the resolution of the output image. The parameter deciding unit 20 bdecides the parameter Pm and the parameter Pn using the resolution ratioSt input from the scaling unit 13 as the enlargement factor.

FIG. 26 is a diagram schematically illustrating a relation between theenlargement factor and the intensity of processing by the low passfilter unit 20 a and the high frequency expanding unit 27.

As the enlargement factor is increased, the jaggy is increased, and thenoise is increased. For this reason, it is desirable to enhanceprocessing by the low pass filter unit 20 a and the high frequencyexpanding unit 27 as the enlargement factor is increased as illustratedin FIG. 26. Here, even when the enlargement factor is 1 (when theinterpolation and the like are not performed), it is desirable toperform the processing weakly instead of stopping all processing by thelow pass filter unit 20 a and the high frequency expanding unit 27 inorder to enhance by removing a fine noise. FIG. 26 illustrates that theenlargement factor is in proportion to the intensity of processing bythe low pass filter unit 20 a and the high frequency expanding unit 27,but the relation may be non-linear.

According to the present embodiment, it is possible to performappropriate image processing even when it is difficult to determineinput resolution based on only terminal information as in the case ofthe HDMI signal in the first embodiment.

Third Embodiment

FIG. 27 is a schematic diagram illustrating a configuration of a displaydevice 3 according to a third embodiment of the present invention.

The display device 3 differs from the display device 1 and the displaydevice 2 in a method of deciding the parameter Pm and the parameter Pn.In the display device 3, the input unit 11 outputs the signal Ssidentifying the input signal to the image processing unit 20, thescaling unit 13 calculates the resolution ratio St, and the calculatedenlargement factor to the image processing unit 20. The parameterdeciding unit 20 b decides the parameter Pm and the parameter Pn basedon the signal Ss input from the input unit 11 and the resolution ratioSt input from the scaling unit 13.

An example of a relation among an input terminal (terminal), inputresolution, and intensity of processing (processing intensity) by thelow pass filter unit 20 a and the high frequency expanding unit 27 inthe present embodiment is shown in Table 2.

TABLE 2 Processing Terminal Input resolution intensity Composite SDStrong Antenna (analog) SD Strong Antenna (digital) HD weak HDMI SD toHD Changed according to enlargement factor

As shown in Table 2, it is possible to determine input resolution ofinput signals from the composite terminal, the antenna (analog)terminal, and the antenna (digital) terminal based on terminalinformation. For this reason, when the signals are selected, theparameter deciding unit 20 b decides intensity of processing based onthe signal Ss input from the input unit 11. On the other hand, it isdifficult to determine the input resolution of the input signal from theHDMI terminal based on only the terminal information. In this case, theparameter deciding unit 20 b decides intensity of processing using theresolution ratio St input from the scaling unit 13 as the enlargementfactor.

In the present embodiment, intensity of processing is decided using theresolution ratio St as the enlargement factor only when it is difficultto determine the input resolution based on the terminal information.According to this configuration, compared to the case of the displaydevice 2 according to the second embodiment, it is possible to reducethe size of a circuit related to image processing and reduce amanufacturing cost.

First Modified Example

Next, in a first modified example of the present embodiment, the samecomponents and processes as in the above embodiment are denoted by thesame reference numerals, and the description proceeds. The displaydevice 1 (see FIG. 1) according to the present modified example includesan image processing unit 30 instead of the image processing unit 20.

FIG. 28 is a schematic diagram illustrating a configuration of the imageprocessing unit 30 according to the present modified example.

The image processing unit 30 is configured to include a contourdirection estimating unit 21, a direction evaluating unit 22, areference region weighting processing unit 23, a pre-processing unit 34,a product-sum operation unit 35, a composition operation unit 36, theparameter deciding unit 20 b, and a high frequency expanding unit 27. Inother words, the image processing unit 30 includes the pre-processingunit 34, the product-sum operation unit 35, and the compositionoperation unit 36 in a low pass filter unit 30 a instead of thepre-processing unit 24, the product-sum operation unit 25, and thecomposition operation unit 26 in the low pass filter unit 20 a of theimage processing unit 20.

The pre-processing unit 34 extracts the brightness signal indicating thesignal value Y(u,v) of each reference pixel (u,v) belonging to thereference region centering on the pixel of interest (i,j) from thebrightness signal Y input from the scaling unit 13 for each pixel ofinterest (i,j). The pre-processing unit 34 subtracts the signal valueY(i,j) of the pixel of interest from the signal value Y(u,v) of thereference signal indicated by the extracted brightness signal, andcalculates the differential signal value Y(u,v)−Y(i,j). Thepre-processing unit 34 generates a differential signal indicating thecalculated differential signal value, and outputs the generateddifferential signal to the product-sum operation unit 35.

The product-sum operation unit 35 receives the direction evaluationvalue information, the reference region weighting information, and thedifferential signal from the direction evaluating unit 22, the referenceregion weighting processing unit 23, and the pre-processing unit 34,respectively, for each pixel of interest (i,j).

The product-sum operation unit 35 calculates a smoothing differentialvalue Y(i,j) by performing the product-sum operation on the differentialsignal value Y(u,v)−Y(i,j) indicated by the differential signal, thedirection evaluation value F(|ΔD|) indicated by the direction evaluationvalue information, and the reference region weighting R(D(i,j),u′,v′)indicated by the reference region weighting information. The product-sumoperation unit 35 uses, for example, Formula (9) to calculate thesmoothing differential value Y(i,j).

$\begin{matrix}{\mspace{79mu}\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 9} \rbrack} & \; \\{{\Delta\;{Y( {i,j} )}} = {\frac{1}{N( {i,j} )}( {\sum\limits_{u^{\prime},v^{\prime}}\;{{F( {{\Delta\; D}} )}{R( {{D( {i,j} )},u^{\prime},v^{\prime}} )}( {{Y( {u,v} )} - {Y( {i,j} )}} )}} )}} & (9)\end{matrix}$

Formula (9) represents that the product of the direction evaluationvalue F(|ΔD|), the reference region weighting R(D(i,j),u′,v′), and thedifferential signal value Y(u,v)−Y(i,j) indicated by the differentialvalue is calculated for each reference pixel, and the sum of thecalculated products for the reference pixels belonging to the referenceregion is calculated. Formula (9) represents that the smoothingdifferential value Y(i,j) is calculated by dividing the calculated sumby the reference area N(i,j). The product-sum operation unit 35generates a smoothing differential signal indicating the calculatedsmoothing differential value Y(i,j), and outputs the generated smoothingdifferential signal to the composition operation unit 36.

The composition operation unit 36 receives the parameter Pm from theparameter deciding unit 20 b, receives the smoothing differential signalfrom the product-sum operation unit 35, and receives the brightnesssignal Y from the scaling unit 13. The composition operation unit 36calculates a low pass signal value Y″(i,j) by performing an addition (acomposition operation) of the product of the smoothing differentialvalue ΔY(i,j) indicated by the smoothing differential signal and theparameter Pm to the signal value Y(i,j) indicated by the brightnesssignal Y for each pixel of interest (i,j). The low pass signal valueY″(i,j) becomes the same value as the low pass signal value Y″(i,j)calculated using Formula (7).

The composition operation unit 36 outputs the brightness signal Y″ basedon the calculated low pass signal value Y″(i,j) to the high frequencyexpanding unit 27.

Next, image processing according to the present modified example will bedescribed.

FIG. 29 is a flowchart illustrating image processing according to thepresent modified example.

The image processing according to the present modified example includessteps S204 to S206 instead of steps S104 to S106 in image processingillustrated in FIG. 24. Step S204 is performed after step S103.

(Step S204) The pre-processing unit 34 extracts the brightness signalindicating the signal value Y(u,v) of each reference pixel (u,v)belonging to the reference region centering on the pixel of interest(i,j) from the brightness signal Y input from the scaling unit 13 foreach pixel of interest (i,j). The pre-processing unit 34 calculates thedifferential signal value Y(u,v)−Y(i,j) based on the extractedbrightness signal (pre-processing). The pre-processing unit 34 outputsthe differential value indicating the calculated differential signalvalue to the product-sum operation unit 35.

Thereafter, the process proceeds to step S205.

(Step S205) The product-sum operation unit 35 receives the directionevaluation value information, the reference region weightinginformation, and the differential signal from the direction evaluatingunit 22, the reference region weighting processing unit 23, and thepre-processing unit 34, respectively, for each pixel of interest (i, j).The product-sum operation unit 35 calculates the smoothing differentialsignal Y(i,j), for example, using Formula (8) based on the differentialsignal value Y(u,v)−Y(i,j) indicated by the differential value, thedirection evaluation value F(|ΔD|) indicated by the direction evaluationvalue information, and the reference region weighting R(D(i,j),u′,v′)indicated by the reference region weighting information. The product-sumoperation unit 35 outputs the smoothing differential signal indicatingthe calculated smoothing differential value Y(i,j) to the compositionoperation unit 36. Thereafter, the process proceeds to step S206.

(Step S206) The composition operation unit 36 calculates the low passsignal value Y″(i,j) by adding the product of the smoothing differentialvalue Y(i,j) indicated by the smoothing differential signal input fromthe product-sum operation unit 35 and the parameter Pm to the signalvalue Y(i,j) indicated by the brightness signal Y input from the scalingunit 13. The composition operation unit 36 outputs the brightness signalY″ indicating the calculated low pass signal value Y″(i,j) to the highfrequency expanding unit 27. Thereafter, the process proceeds to stepS107.

As described above, in the present modified example, the compositionsignal value of each pixel is calculated using the value obtained byperforming the product-sum operation on the differential signal valuebased on the signal value of each reference pixel between the referencepixels that are in the direction of the predetermined range from thecontour direction of each pixel. As a result, it is possible to reduce aprocessing amount without undermining the effect of removing or reducingthe noise of the visually sensitive contour direction.

Second Modified Example

Next, in a second modified example of the present embodiment, the samecomponents and processes as in the above embodiment are denoted by thesame reference numerals, and the description proceeds. The displaydevice 1 (see FIG. 1) according to the present modified example includesan image processing unit 40 instead of the image processing unit 20.

FIG. 30 is a schematic diagram illustrating a configuration of the imageprocessing unit 40 according to the present modified example.

The image processing unit 40 is configured to include a contourdirection estimating unit 21, a direction evaluating unit 22, areference region weighting processing unit 43, a pre-processing unit 34,a product-sum operation unit 35, a composition operation unit 36, theparameter deciding unit 20 b, and a high frequency expanding unit 27. Inother words, the image processing unit 40 includes the reference regionweighting processing unit 43 in a low pass filter unit 40 a instead ofthe reference region weighting processing unit 23 in the low pass filterunit 30 a of the image processing unit 30.

The reference region weighting processing unit 43 decides the weightingcoefficient R(D(i,j),u′,v′) based on the quantization contour directionD(i,j) of each pixel indicated by the quantization contour directioninformation input from the contour direction estimating unit 21,similarly to the reference region weighting processing unit 23. Thereference region weighting processing unit 43 selects the weightingcoefficient R(D(i,j),u′,v′) of each reference pixel (u′,v′) in which(D(i,j),u′,v′) has a non-zero value among the weighting coefficientsR(D(i,j),u′,v′). The reference pixel is positioned in the contourdirection or in the direction approximating to the contour directionfrom the pixel of interest (i,j) and thus referred to as a contourdirection reference pixel. The reference region weighting processingunit 43 generates the reference region weighting information indicatingthe weighting coefficient R(D(i,j),u′,v′) related to each contourdirection reference pixel, and outputs the generated reference regionweighting information to the product-sum operation unit 35.

The reference region weighting processing unit 43 extracts thequantization contour direction information indicating the quantizationcontour direction D(u,v) related to each contour direction referencepixel from the input quantization contour direction information. Thereference region weighting processing unit 43 outputs the extractedquantization contour direction information to the direction evaluatingunit 22.

The reference region weighting processing unit 43 extracts thebrightness signal indicating the signal value Y(u,v) related to eachcontour direction reference pixel from the brightness signal input fromthe scaling unit 13. The reference region weighting processing unit 43outputs the extracted brightness signal to the pre-processing unit 34.

The pre-processing unit 34, the direction evaluating unit 22, and theproduct-sum operation unit 35 perform the product-sum operation on thedirection evaluation value F(|ΔD|), the weighting coefficientR(D(i,j),u′,v′), and the differential value Y(u,v)−Y(i,j) to calculatethe smoothing differential value Y(i,j) for each contour directionreference pixel. Here, the product-sum operation unit 35 uses Formula(10) instead of Formula (9).

$\begin{matrix}{\mspace{79mu}\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 10} \rbrack} & \; \\{{\Delta\;{Y( {i,j} )}} = {\frac{1}{N( {i,j} )}( {\sum\limits_{u^{\prime},{v^{\prime} \in {{Rs}{({D{({i,j})}})}}}}\;{{F( {{\Delta\; D}} )}{R( {{D( {i,j} )},u^{\prime},v^{\prime}} )}( {{Y( {u,v} )} - {Y( {i,j} )}} )}} )}} & (10)\end{matrix}$

In Formula (10), R_(s)(D(i,j)) indicates a function (a region selectionfunction) of selecting the contour direction reference pixel among thereference pixels related to the pixel of interest (i, j). In otherwords, u′,v′εRs (D(i,j)) indicates the contour direction referencepixel. Thus, the product-sum operation unit 35 can calculate thesmoothing differential value Y(i,j) equal to the smoothing differentialvalue Y(i,j) calculated when Formula (9) is used.

As described above, the reference region weighting processing unit 43extracts the signal value or the coefficient related to each contourdirection reference pixel and thus can reduce a computation amount inthe pre-processing unit 34, the direction evaluating unit 22, and theproduct-sum operation unit 35 while obtaining the same operation resultas in the first modified example.

Next, the image processing according to the present modified examplewill be described.

FIG. 31 is a flowchart illustrating the image processing according tothe present modified example.

The image processing according to the present modified example includesstep S303 instead of step S103 in image processing illustrated in FIG.29. Step S303 is performed after step S101. Step S204 is performed afterstep S102. Steps S204 to S206 are performed on each contour directionreference pixel instead of each reference pixel of each pixel ofinterest.

(Step S303) The reference region weighting processing unit 43 decidesthe weighting coefficient R(D(i,j),u′,v′) for each pixel based on thequantization contour direction information input from the contourdirection estimating unit 21. The reference region weighting processingunit 43 selects a weighting coefficient R(D(i,j),u′,v′) having anon-zero value among the decided weighting coefficients R(D(i,j),u′,v′)as the weighting coefficient R(D(i,j),u′,v′) of each contour directionreference pixel. The reference region weighting processing unit 43outputs the reference region weighting information indicating theselected weighting coefficient R(D(i,j),u′,v′) to the product-sumoperation unit 35. The reference region weighting processing unit 43extracts the quantization contour direction information indicating thequantization contour direction D(u,v) related to each contour directionreference pixel from the input quantization contour directioninformation, and outputs the extracted quantization contour directioninformation to the direction evaluating unit 22. The reference regionweighting processing unit 43 extracts the brightness signal indicatingthe signal value Y(u,v) related to each contour direction referencepixel from the brightness signal input from the scaling unit 13, andoutputs the extracted brightness signal to the pre-processing unit 34.Thereafter, the process proceeds to step S102.

As described above, in the present modified example, the weightingcoefficient, the evaluation value, and the signal value of eachreference pixel that is in the direction of the predetermined range fromthe contour direction of each image are extracted, and the product-sumoperation is performed using the extracted weighting coefficient, theevaluation value, and the signal value of each reference pixel. Thus,the weighting coefficient, the evaluation value, and the signal valuethat do not contribute to the operation result are excluded. Thus, it ispossible to reduce a processing amount related to an operation and thestorage capacity of the storage unit without undermining the effect ofremoving or reducing the noise of the visually sensitive contourdirection with the relative small computation amount.

Third Modified Example

Next, in a third modified example of the present embodiment, the samecomponents and processes as in the above embodiment are denoted by thesame reference numerals, and the description proceeds. The displaydevice 1 (see FIG. 1) according to the present modified example includesa high frequency expanding unit 37 instead of the high frequencyexpanding unit 27 in the image processing unit 20 (FIG. 2), the imageprocessing unit 30 (FIG. 28), or the image processing unit 40 (FIG. 30).

FIG. 32 is a schematic diagram illustrating a configuration of the highfrequency expanding unit 37 according to the present modified example.

The high frequency expanding unit 37 is configured to include anon-linear filter unit 38 and a composition operation unit 39.

The non-linear filter unit 38 is configured to include a 2D high passfilter unit 381 and a non-linear operation unit 382.

The 2D high pass filter unit 381 receives the brightness signal Y″ andthe quantization contour direction information from the compositionoperation unit 26 (or the composition operation unit 36) and the contourdirection estimating unit 21, respectively. The 2D high pass filter unit381 calculates a contour direction component signal W_(2D) indicating ahigh-pass component related to the quantization contour direction D(i,j)indicated by the quantization contour direction information for the lowpass signal value Y″(i,j) indicated by the brightness signal Y″. The 2Dhigh pass filter unit 381 outputs the calculated contour directioncomponent signal W_(2D) to the non-linear operation unit 382. Aconfiguration of the 2D high pass filter unit 381 will be describedlater.

The non-linear operation unit 382 has the same configuration as thenon-linear operation unit 282-A or 282-B. The non-linear operation unit382 performs the non-linear operation on the signal value indicated bythe direction component signal W_(2D) input from the 2D high pass filterunit 381. The non-linear operation unit 382 outputs the high frequencycomponent value NL_(2D) indicated by the calculated non-linear outputvalue to the composition operation unit 39.

The composition operation unit 39 has the same configuration as thecomposition operation unit 29. The composition operation unit 39calculates the high frequency extension signal value Z(i,j) by adding(synthesizing) the low pass signal value Y″(i,j) to the product of thehigh frequency component value NL_(2D) input from the non-linearoperation unit 382 and the parameter Pn. The composition operation unit39 generates the brightness signal Z indicating the calculated highfrequency extension signal value Z(i,j), updates the brightness signal Yinput from the scaling unit 13 to the brightness signal Z, andsynthesizes the brightness signal Z with the color-difference signals Cband Cr. The composition operation unit 39 outputs an image signalincluding the brightness signal Z and the color-difference signals Cband Cr to the image format converting unit 14.

Next, a configuration of the 2D high pass filter unit 381 will bedescribed.

FIG. 33 is a schematic diagram illustrating a configuration of the 2Dhigh pass filter unit 381 according to the present modified example.

The 2D high pass filter unit 381 is configured to include a delay memory3811, a direction selecting unit 3812, a multiplying unit 3813, a filtercoefficient memory 3814, and a composition operation unit 3815.

The delay memory 3811 includes delay elements 3811-1 to 3811-2 n+1 thatdelay an input signal by 2n+1 W_(x) samples. The delay elements 2811-v−1to 2811-v−2n+1 output delay signals including signals value of 2n+1samples obtained by delaying the input signal by W_(x)−2n, W_(x)−2n+1, .. . , and W_(x) samples to the direction selecting unit 3812.

The delay elements 3811-1 to 3811-2 n+1 are connected in series. One endof the delay element 3811-1 receives the low pass signal value Y″(i,j)indicated by the brightness signal Y″, and the other end of the delayelement 3811-1 outputs the delay signal obtained by delaying by W_(x)samples to one end of the delay element 3811-2. One ends of the delayelement 3811-2 to 3811-2 n+1 receive the delay signals obtained bydelaying by W_(x) to 2n·W_(x) samples from the other ends of the delayelements 3811-1 to 3811-2 n. The other end of the delay element 3811-2n+1 outputs the delay signal obtained by delaying (2n+1)·W_(x) samplesto the direction selecting unit 3812. Thus, the signal values of(2n+1)·(2n+1) pixels neighboring one another in the horizontal directionand the vertical direction centering on the pixel of interest, whichindicate the brightness signal Y″, are output to the direction selectingunit 3812.

The pixels corresponding to the signal values are the reference pixelsbelonging to the reference region centering on the pixel of interest.

The direction selecting unit 3812 selects the reference pixel (u′,v′)that is in the quantization contour direction D from the pixel ofinterest (i,j) or the direction approximating to the quantizationcontour direction D based on the quantization contour direction D(i,j)of each pixel indicated by the quantization contour directioninformation input from the contour direction estimating unit 21. Forexample, a reference pixel to be selected is a reference pixelsatisfying the following condition: (1) a reference pixel (u′,v′)through which a line segment extending from the center of the pixel ofinterest (i,j) in the quantization contour direction passes; (2) alength in the horizontal direction or the vertical direction by whichthe line segment passes through is 0.5 pixel or larger; and (3) onereference pixel is selected for each of 2n+1 coordinates in at least oneof the horizontal direction and the vertical direction. Hereinafter, theselected reference pixel is also referred to as a selection referencepixel.

The direction selecting unit 3812 determines whether a direction(hereinafter, also referred to as a “selection coordinates direction”)in which each reference pixel is selected is the horizontal direction,the vertical direction, or both of the horizontal direction and thevertical direction for each of 2n+1 coordinates. For example, thedirection selecting unit 3812 outputs signal values related to the 2n+1selection reference pixels to the multiplying unit 3813 in thedescending order of the indices of the selection coordinates directions.

The direction selecting unit 3812 may include a storage unit in whichselection reference pixel information indicating the selection referencepixel of each quantization contour direction is stored in advance andselect the selection reference pixel information corresponding to thequantization contour direction D(i,j) of the pixel of interest (i,j). Asa result, the direction selecting unit 3812 outputs the signal valuerelated to the selection reference pixel indicated by the selectedselection reference pixel information to the multiplying unit 3813.

Next, an exemplary selection reference pixel will be described.

FIG. 34 is a conceptual diagram illustrating an exemplary selectionreference pixel.

A relation between a horizontal axis and a vertical axis and arrowsindicating quantization contour directions of FIG. 34 are the same as inFIG. 17.

Each of grid-like diagrams in directions in which arrows indicated bynumerical values 0 to 7 counterclockwise from the right side of anoriginal point move toward an upper half plane indicates selectionreference pixel R_(s) (D(i,j),u′,v′) corresponding to the quantizationcontour direction D(i,j). In FIG. 34, the number of reference images is7 in the horizontal direction and 7 in the vertical direction. Anumerical value shown in each quadrangle included in the grid-likediagram is a numerical value R_(s) (0,u′,v′) indicating whether or noteach pixel is the selection reference pixel. 1 indicates that a pixel isthe selection reference pixel, and 0 indicates that a pixel is not theselection reference pixel.

For example, the selection reference pixels R_(s) (0,u′,v′)corresponding to the quantization contour direction 0 are all referencepixels in a fourth column. The reference pixels in the other columns arenot the selection reference pixel. In other words, the selectioncoordinates direction for the quantization contour direction 0 is thevertical direction. The selection reference pixels corresponding to thequantization contour direction 5 are reference pixels in a first columnof a fifth row, a second column of the fifth row, a third column of afourth row, a fourth column of the fourth row, a fifth column of thefourth row, a sixth column of a third row, and a seventh column of thethird row. In other words, the selection coordinates direction for thequantization contour direction 5 is a direction that is closer to thehorizontal direction than the vertical direction and oblique to theright.

Referring back to FIG. 33, the multiplying unit 3813 is configured toinclude 2n+1 multipliers 3813-1 to 3813-2 n+1. The multipliers 3813-1 to3813-2 n+1 multiply the signal values input from the direction selectingunit 3812 by the filter coefficients read from the filter coefficientmemory 3814, and output multiplication values to the compositionoperation unit 3815. Here, the signal values are input so that the orderof the multipliers 3813-1 to 3813-2 n+1 matches the order (thedescending order of the indices of the selection coordinates directions)of the signal values input thereto.

The filter coefficient memory 3814 stores 2n+1 filter coefficientsa_(D−n), a_(D−n+1), . . . , and a_(D+n) used in the multipliers 3813-1to 3813-2 n+1 in advance. The filter coefficients a_(D−n), a_(D−n+1), .. . , and a_(D+n) are high-pass filter coefficients for implementing thehigh pass filter through the product-sum operation with the signalvalue. The high-pass filter coefficient may have a value that has thesame high pass characteristics as the filter coefficients a_(D−n),a_(D−n+1), . . . , and a_(D+n) and has characteristics of blocking a DCcomponent. The composition operation unit 3815 adds the 2n+1multiplication values input from the multiplying unit 3813, andgenerates a contour direction component signal W_(2D) having a signalvalue serving as the sum of the 2n+1 multiplication values. Thecomposition operation unit 3815 outputs the generated contour directioncomponent signal W_(2D) to the non-linear operation unit 382.

Next, image processing according to the present modified example will bedescribed.

FIG. 35 is a flowchart illustrating image processing according to thepresent modified example.

The image processing according to the present modified example includesstep S407 instead of step S107 in the image processing illustrated inFIG. 31.

(Step S407) The high frequency expanding unit 37 calculates the contourdirection component signal W_(2D) indicating the high-pass component forthe quantization contour direction D(i,j) indicated by the quantizationcontour direction information input from the contour directionestimating unit 21 with respect to the brightness signal Y″ input fromthe composition operation unit 26 (or the composition operation unit36). The high frequency expanding unit 37 performs the non-linearoperation on the signal value indicated by the contour directioncomponent signal W_(2D), and calculates the high frequency componentvalue NL_(2D) indicating the high frequency component. The highfrequency expanding unit 37 synthesizes the product of the highfrequency component value NL_(2D) and the parameter Pn with the low passsignal value Y″(i,j) indicated by the brightness signal Y″, andcalculates the high frequency extension signal value Z(i,j). The highfrequency expanding unit 27 outputs the brightness signal indicating thecalculated high frequency extension signal value Z(i,j) to the imageformat converting unit 14. Thereafter, the process ends.

As described above, in the present modified example, the high frequencycomponent of the contour direction of each image of interest isgenerated, and the generated high frequency component is synthesizedwith the low-pass component smoothed in the contour direction. As aresult, it is possible to remove or reduce noise in a visually sensitivecontour direction and sharpen an image extending in the tangentdirection vertical to the contour direction.

(Exemplary Processing)

Next, an exemplary image generated by performing processing according tothe present embodiment will be described.

The following description will proceed with an example in which thevalue of the parameter Pn is 1, and the value of the parameter Pm is 1.

FIG. 36 illustrates exemplary images (images 86 to 88) related tobrightness signals before and after processing is performed.

The image 86 is an image indicated by a brightness signal obtained byscaling a brightness signal related to a low resolution image with atriple enlargement factor. Step-like noise (jaggy) occurs in theboundary between a bright region and a dark region of the image 86, andan unclear boundary (blur) is shown.

The image 87 is an image indicated by a brightness signal obtained byperforming the processing according to the example of the presentembodiment on the brightness signal related to the image 86. In theimage 87, the boundary between the bright region and the dark region issmooth, and the step-like noise is removed, and the boundary is clearlyshown.

The image 88 is an image indicated by a brightness signal obtained byperforming a high-pass component emphasis process performed in therelated art on the brightness signal related to the image 86. In theimage 88, the boundary between the bright region and the dark region isclearer than in the image 86, but the step-like noise shown in the image86 is also clearly shown.

As a result, it is clearly shown that the noise is reduced by theprocessing according to the present embodiment, and the image issharpened.

FIG. 37 illustrates exemplary spatial frequency characteristics (images89 to 92) of an image before and after processing is performed.

In the images 89 to 92, a left-right direction indicates a spatialfrequency in the horizontal direction, and an up-down directionindicates a spatial frequency in the vertical direction. In the images89 to 92, the center indicates a zero spatial frequency, that is, a DCcomponent. The images 89 to 92 are images in which the level in eachspatial frequency is indicated by the contrasting density. A level islarger in a bright portion, and a level is smaller in a dark portion.

The image 89 shows a spatial frequency characteristic of a lowresolution image. In the image 89, dots distributed from an upper rightportion to a lower left portion on an oblique line indicate thatcorresponding spatial frequency components are included in the lowresolution image.

The image 90 is an image indicating the spatial frequency characteristicof the image 86. In the image 90, dots distributed from an upper rightportion to a lower left portion on an oblique line are shown, but no dotis shown in portions surrounded by dotted lines at upper and lower ends.This represents that the high-pass component is blocked in the scalingprocess for generating the image 86. A lack of the high-pass componentcauses the boundary to be unclear. Meanwhile, dots are shown in portionssurrounded by alternated long and short dash lines of the image 90 atboth left and right sides. This represents that aliasing occurs with thescaling. The aliasing corresponds to the step-like noise in the image86.

The image 91 is an image indicating a spatial frequency characteristicof the image 87, that is, an image obtained by performing the processingaccording to the present embodiment. In the image 91, the same dots asin the image 89 are shown even in portions surrounded by the dottedlines. This represents that the lack of the high-pass component in theimage 86 is compensated, and the boundary in the image 87 becomes clear.In the image 91, no dot is shown in the same frequency regions as theportions surrounded by the alternated long and short dash lines of theimage 90. This represents that the step-like noise shown in the image 86has been removed.

The image 92 is an image indicating a spatial frequency characteristicof the image 86, that is, an image obtained by the high-pass componentemphasis process. In portions surrounded by alternated long and shortdash lines at the left and right sides of the image 92, dots are shownmore noticeably than in the image 90. This represents that as thehigh-pass component is emphasized, aliasing is also emphasized, and thestep-like noise is clearly shown in the image 88.

Next, exemplary data used or generated in the present embodiment will bedescribed.

FIG. 38 illustrates spatial frequency characteristic (images 93 to 97)of data used or generated in the present embodiment.

In the images 93 to 97, a left-right direction indicates a spatialfrequency in the horizontal direction, and an up-down directionindicates a spatial frequency in the vertical direction. In the images93 to 97, the center indicates a zero spatial frequency, that is, a DCcomponent. In the images 94, 96, and 97, a level is large in a brightportion, and a level is small in a dark portion.

The image 93 is an image indicating filter characteristics of the lowpass filter unit 20 a for the brightness signal Y. In the image 93, ashaded region is a blocking region, and a white region is a passingregion. The passing region is a strip-like region extending from anupper right to a lower left. Thus, the low pass filter unit 20 atransmits a frequency component lower than a blocking frequency amongspatial frequency components corresponding to the contour direction, andtransmits spatial frequency components corresponding to the otherdirections without band limitation. Through this operation, the low passfilter unit 20 a performs smoothing on the signal values in the contourdirection. The blocking frequency may differ according to the selecteddirection evaluation region weighting but is averagely smaller thanf_(s)/(2·n) when the size of the reference region (the number of pixelsin one direction) is decided to be 2n+1. f_(s) is a sampling frequencyof a brightness signal of a processing target, and f_(s)/(2·n)corresponds to a Nyquist frequency f_(nyq)′ of a brightness signalbefore enlarged.

The image 94 is an image indicating a spatial frequency characteristicof the brightness signal Y″ indicated by the low pass signal valueY″(i,j) output from the low pass filter unit 20 a. In the image 94, dotspassing through the central portion of the image 90 are shown, but dotsat both left and right sides are not shown. This represents that thespatial frequency component corresponding to the contour direction hasbeen extracted by the low pass filter unit 20 a. It represents that thestep-like noise occurring by the aliasing has been consequently removed.

The image 95 is an image indicating filter characteristics in which thefilter characteristics of the horizontal high pass filter unit 281-h issynthesized with the filter characteristics of the vertical high passfilter unit 281-v. In the image 93, a shaded region is a blockingregion, and a white region is a passing region. The passing region is aframe-like region surrounding the blocking region of a quadranglecentering an original point.

The image 96 is an image indicating a spatial frequency characteristicof the direction component signal W obtained by causing the low passsignal value Y″(i,j) to pass through the horizontal high pass filterunit 281-h and the vertical high pass filter unit 281-v. The dots shownin the image 94 are not shown in a central portion of the image 96. Thisrepresents that the components of the passing band indicated by theimage 95 among the spatial frequency components indicated by the image94 has been extracted.

The image 97 is an image indicating a spatial frequency characteristicof the high frequency component value NL_(A) obtained by inputting thedirection component signal W to the non-linear operation unit 282-A.Here, unclear dots that are not shown in the image 96 and are unclearnear the upper end and the lower ends are clearly shown in the image 97.This represents that a high frequency component for the directionvertical to the contour direction has been generated. The spatialfrequency characteristic indicated by the image 91 corresponds to whatthe spatial frequency characteristic indicated by the image 97 and thespatial frequency characteristic indicated by the image 94 arecomplementary, and approximates to the spatial frequency characteristicof the image 89. Thus, in the present embodiment, it is proved that itis possible to sharpen an image while reducing noise by synthesizing thelow pass signal value Y″(i,j) with the high frequency component valueNL_(A) and generating the high frequency extension signal value Z(i,j).

Next, an exemplary high frequency component generated through thenon-linear operation will be described.

Here, an output frequency of an output value y obtained by executing afunction f(x) for a sine wave of an input frequency serving as an inputvalue x is illustrated.

FIG. 39 is a diagram illustrating an exemplary output frequency by anon-linear operation.

In FIG. 39, a vertical axis denotes an input frequency, and a horizontalaxis denotes an output frequency. FIG. 39 illustrates an outputfrequency obtained by using the function f(x)=sgn (x)|x|² when the inputfrequency is changed from 0 to f_(s)/4. f_(s) is a sampling frequency.In FIG. 39, the input frequency is a thick sold line, and a frequency ofa component having the highest level among the output values isindicated by a thick dotted line. The thick dotted line represents thatthe output frequency changes from 0 to 3·f_(s)/4 when the inputfrequency is changed from 0 to f_(s)/4. In other words, a tripleharmonic that is three times as high as an input frequency is mainlyoutput. Thus, it is represented that a high frequency component that isnot included in the input value x is obtained by executing the functionf(x) on the input value x.

FIG. 40 is a diagram illustrating another exemplary output frequency bya non-linear operation.

A vertical axis, a horizontal axis, and an input frequency of FIG. 40are the same as in FIG. 39. Here, FIG. 40 illustrates an outputfrequency obtained using the function f(x)=|x|³. In FIG. 40, an outputfrequency is indicated by a thick dotted line. The thick dotted linerepresents that the output frequency changes from 0 to 3·f_(s)/4 whenthe input frequency is changed from 0 to f_(s)/4. Only a triple harmonicthat is three times as high as an input frequency is output. Similarlyto the example illustrated in FIG. 39, a high frequency component thatis not included in the input value x is obtained by executing thefunction f(x) on the input value x, but a component other than a tripleharmonic is not output.

In the above embodiment, the antenna is not limited to a radio waverelated to television broadcasting and may receive a radio wave relatedto public wireless communication.

The above description has proceeded under the assumption that a colorsystem of a generated image signal is a YCbCr color system. When agenerated image signal is indicated by a color system configured withsignal values indicating brightness of respective colors (for example,an RGB color system), the image processing unit 20 may performprocessing on signal values of respective colors.

In the above embodiment, by the contour direction estimating unit 21,the contour directions θ (before quantization) calculated for respectivepixels may be averaged within an image block including a predeterminednumber (for example, 3 in the horizontal direction and 3 in the verticaldirection, that is, a total of 9) of neighboring pixels centering on thepixel of interest. The contour direction estimating unit 21 quantizesthe averaged contour direction. As a result, it is possible to smooth anerror locally and notably occurring in the contour direction betweenpixels and reproduce a natural image as a whole.

In the above embodiment, in the differential filter, the range of thedifferential direction (the x direction) in which the filter coefficientW_(x)(uy,vy) is 1 and the range of the differential direction in whichthe filter coefficient W_(x)(u′,v′) is −1 may not be the same in thedirection (the y direction) vertical to the differential direction. Whenthe range of the differential direction in which the filter coefficientW_(x)(u′,v′) is 1 and the range of the differential direction in whichthe filter coefficient W_(x)(u′,v′) is −1 are symmetric with respect tou′=0, and v′ is 0, it is preferable that the ranges be n or larger. Forexample, the range of the differential direction in which the filtercoefficient W_(x)(u′,v′) is 1 may be n when v′=0 and may be smaller thann when v′≠0.

Similarly, the range of the differential direction (the y direction) inwhich the filter coefficient W_(y)(u′,v′) is 1 and the range of thedifferential direction in which the filter coefficient W_(y)(u′,v′) is−1 may not be the same in the direction (the x direction) vertical tothe differential direction. When the range of the differential directionin which the filter coefficient W_(y)(u′,v′) is 1 and the range of thedifferential direction in which the filter coefficient W_(y)(u′,v′) is−1 are symmetric with respect to v′=0, and u′ is 0, it is preferablethat the ranges be n or larger. For example, the range of thedifferential direction in which the filter coefficient W_(y)(u′,v′) is 1may be n when u′=0 and may be smaller than n when u′≠0.

In the above embodiment, some components of the display device 1, forexample, the input unit 11, the scaling unit 13, the image formatconverting unit 14, and the image processing units 20, 30, and 40 may beimplemented by a computer. In this case, a program for implementing thiscontrol function is recorded in a computer readable recording medium,and the program may be implemented such that the program recorded in therecording medium is read and executed by a computer system. Here, the“computer system” is assumed to be a computer system installed in thedisplay device 1 and include an operating system (OS) and hardware suchas a peripheral device. The “computer readable recording medium” refersto a storage device such as a flexible disk, a magneto optical disc,read only memory (ROM), a portable medium such as a CD-ROM, or a harddisk installed in a computer system. The “computer readable recordingmedium” may also include a medium holding a program dynamically during ashort period of time such as a communication line when a program istransmitted via a network such as the Internet or a communication linesuch as a telephone line and a medium holding a program for a certainperiod of time such as a volatile memory in a computer system serving asa server or a client in this case. The program may implement somefunctions among the above-described functions and may implement theabove-described functions in combination with a program previouslystored in a computer system.

In the above embodiment, some or all components of the display device 1may be implemented as integrated circuits (ICs) such as large scaleintegration (LSI). Each of the functional blocks of the display device 1may be implemented as a processor, and all or some of the functionalblocks may be integrated and implemented as a processor. An IC techniqueis not limited to the LSI, and implementation may be performed by adedicated circuit or a general-purpose processor. Further, when an ICtechnique replacing the LSI is developed with the advance ofsemiconductor technology, an IC by such technique may be used.

One embodiment of the invention have been described above in detail withreference to the appended drawings, but a concrete constitution is notlimited to the above embodiment, and various design changes or the likecan be made within the scope not departing from the gist of theinvention.

The invention claimed is:
 1. An image processing device, comprising: aplurality of input terminals to which an input image signal is input;contour direction estimating circuitry that estimates a contourdirection in which a signal value is a constant value for each of aplurality of pixels; a low pass filter that smoothes the signal value ofa pixel of the plurality of pixels based on the signal value of eachreference pixel serving as a pixel of a reference region according tothe pixel and arranged in the contour direction of the pixel estimatedby the contour direction estimating circuitry for each of the pluralityof pixels; and parameter deciding circuitry that decides an intensity ofsmoothing by the low pass filter, wherein the parameter decidingcircuitry decides the intensity of smoothing based on a type of theinput terminal when a resolution of the input image signal is determinedto be unique based on the type of the input terminal to which the inputimage signal is input among the plurality of input terminals, and theparameter deciding circuitry decides the intensity of smoothing based ona resolution ratio that is a ratio of resolution of an input imagesignal and resolution of an output image signal when the resolution ofthe input image signal is not determined to be unique based on the inputterminal to which the input image signal is input among the plurality ofinput terminals.
 2. The image processing device according to claim 1,further comprising: scaling circuitry that performs interpolation of theinput image signal based on the resolution ratio, and outputs theresolution ratio to the parameter deciding circuitry.
 3. The imageprocessing device according claim 1, further comprising: high frequencyexpanding circuitry that generates a high frequency component of thesignal value of the pixel for each of the plurality of pixels, andexpands a frequency band for the signal value of the pixel, wherein theparameter deciding circuitry decides intensity of processing by the highfrequency expanding circuitry based on the type of the input terminalwhen the resolution of the input image signal is determined to be uniquebased on the type of the input terminal to which the input image signalis input among the plurality of input terminals, and the parameterdeciding circuitry decides the intensity of processing by the highfrequency expanding circuit based on the resolution ratio when theresolution of the input image signal is not determined to be uniquebased on the input terminal to which the input image signal is inputamong the plurality of input terminals.